Friday, 15 August 2014

Installing counterpartyd

There are some docs out-there to talk you through it (README.md), but it felt like a bit of a pain... Here is my 'concise' version with notes...

I'm running Mint on wheezy/sid
Kernel: 3.11.0-12-generic x86_64 (64 bit) Desktop: Gnome Distro: Linux Mint 16 Petra

BTC

I already had bitcoind up and running (it came with the distro), i.e. see:
apt-cache search bitcoin

(or http://counterpartyd-build.readthedocs.org/en/latest/SettingUpBitcoind.html)

In my case this installed the blockchain under ~/.bitcoin (I'm not sure if I configured that, or if that's where bitcoin-qt puts it by default).

In that location (~/.bitcoin), here is what I have in bitcoin.conf:
rpcuser=bitcoinrpc
rpcpassword=whatever99
txindex=1
server=1

Note that bitcoind is a daemon you run on your own machine (hence the term 'node'). The daemon accepts remote procedure calls (something like API calls), for example:
bitcoind getblockcount

As far as I understand, the rpc user and password settings are what the daemon uses to restrict such calls. As counterpartyd needs to talk to bitcoind, we'll need to pass that password to counterpartyd later.

The tindex line indexes the blockchain as it's downloaded, and is required for functionality used by counterpartyd.

The server line allows bitcoind to receive rpc calls.

You can add testnet=1 to run on the bitcoin testnet.

While running bitcoind, the network, memory and CPU usage of my computer went through the roof. This is expected, as the bitcoin blockchain is first downloaded, verified and indexed. It's a one off thing that settles down once you're back at 1 block every 10 minutes. Note that bitcoin-qt does exactly the same thing... literally. i.e. don't run bitcoin-qt at the same time as bitcoind, because they both try to download the blockchain to the same location!

XCP

I grabbed counterpartyd from git and switched to the (currently) latest tag:
git clone  https://github.com/CounterpartyXCP/counterpartyd.git
cd counterpertyd; git checkout  v9.39.0

I installed all the python requirements using virtualenv like this:
virtualenv MyVE --python python3
source MyVE/bin/activate
pip install --upgrade -r pip-requirements.txt

I found that I needed to use apt to get some development libraries:
sudo apt-get install python3-dev

Running ./counterpartyd.py --help showed that I also needed apsw. Trying to install that with pip failed due to first missing and then out of date libsqlite3-dev.

So... outside the virtualenv (for no good reason) I grabbed apsw from git:
git clone https://github.com/rogerbinns/apsw.git

Fetched the latest sqlite3 (using my virtualalenv):
~/counterpartyd/MyVE/bin/python setup.py fetch --sqlite --missing-checksum-ok

And finally installed apsw (using my virtualalenv):
~/counterpartyd/MyVE/bin/python setup.py install

Now at least ./counterpartyd.py --help worked, but figuring out the right settings for 'rpc' was a pain... The documents refer to '--bitcoind-rpc-password', which I dutifully set in a file called counterpartyd.conf. Naturally, I set it to match the rpcpassword in bitcoin.conf. However, running something like ./counterpartyd.py server (which is what you should run to start the counterparty daemon) complained about --backend-rpc-password not being set, which was confusing.

It seems that the 'backend' is bitcoind, so I should have done this:
./counterpartyd.py --backend-rpc-password=whatever99 server

Confusingly this then complains about --rpc-password not being set. You can just set this to 'whaever101', as it's just the counterpartyd equivalent of the bitcoind rpc-password. 

So to be clear (because it confuses me):
  • bitcoind uses something called rpcpassword to restrict access, and you set it to whatever you want when you start the deaemon (e.g. whatever99).
  • counterpartyd needs to know that password, but it calls it backend-rpc-password
  • counterpartyd uses something called rpc-password to restrict access, and you set it to whatever you want when you start the deaemon (e.g. whatever101).
As far as I could tell everything I tried setting in counterpartyd.conf was ignored. All the above was confusing to work out because of verbose error messages and confusing 'connection' errors when you got things wrong.

So finally:
./counterpartyd.py --backend-rpc-password=whatever99 --rpc-password=whatever101 server

The first thing this does (analogously to bitcoind) is parse through the blockchain and pull out all valid counterparty 'actions' into a database.

It's fun to watch the burn reply:
Burn: 16kNtJqcAqKDgpTGor2jYjahxLn1khiony burned 1.0 BTC for 1467.72727273 XCP (ef597b82df3afde4236f0d0e032160d02af31ac96cf28dc10217fb73a47987ef) [valid]

It's strange to think that setting fire to your money could be a sound investment...

Well... Now I'm stuck waiting for the index to build...


Friday, 21 March 2014

Pester pester pester...


Date: Tue, 13 Oct 2009 19:08:45 +0100
Subject: Request for data "Genetic Determinants of Financial Risk Taking"
From: Dan Bolser
To: c-kuhnen@kellogg.northwestern.edu, jchiao@northwestern.edu

Dear Drs. Kuhnen and Chiao,

I read with great interest your article "Genetic Determinants of
Financial Risk Taking".

I noticed that the p-values for the significance of the difference in
the means (figure 1 B and C) are quite marginal, less than 0.02 and
less than 0.04 for 5-HTTLPR and DRD4 respectively.

Was there a significant difference under a two-tailed significance test?

I would be interested to see the underlying data so that I can try my
own statistical analysis. I'm reviewing your paper for a journal club,
and I think one question for discussion will be the difference between
a one-tailed and a two-tailed significance test. I note that your
hypothesis for the direction of the difference seems valid, and that a
one-tailed test seems appropriate in this case.

One question that I would love to know with regard to the work, which
was missing from the paper... Which group of people made the most
money during the test? I guess this result is somewhat 'sensitive'?
;-)

Thanks very much for your assistance with the above request and the questions.

I've shown this paper to about 10 people so far, and I'm really
looking forward to reviewing it. I think it touches on one of the
"defining issues of our age" (to steal a piece of hyperbole from the
global warming campaigners... actually I think you could go so far as
to call your results "an inconvenient truth"!).

Sincerely,
Dan Bolser.



Date: Tue, 13 Oct 2009 13:37:59 -0500
From: "Camelia M. Kuhnen"
To: Dan Bolser
Cc: jchiao@northwestern.edu
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"

Dear Dan,
Thank you for your interest in our paper. Unfortunately, we are unable
to share the individual level data from this project at this point.
Regards,
Camelia

Camelia M. Kuhnen
Assistant Professor of Finance
Kellogg School of Management
Northwestern University
2001 Sheridan Road
Evanston, IL 60208-2001

E-mail: c-kuhnen@kellogg.northwestern.edu
Phone: 847-467-1841
Fax:   847-491-5719
Web: http://www.kellogg.northwestern.edu/faculty/kuhnen/htm/index.html



Date: Wed, 14 Oct 2009 02:19:51 +0100
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"
From: Dan Bolser
To: "Camelia M. Kuhnen"
Cc: jchiao@northwestern.edu

2009/10/13 Camelia M. Kuhnen :
> Dear Dan,
> Thank you for your interest in our paper. Unfortunately, we are unable to
> share the individual level data from this project at this point.

Why is that if you don't mind me asking?

At what point do you think you will be able to share it?

I was wondering, for example, if there were any significant
associations with gender that could be used (on average) to improve
the calculation of the residual 'excess risky investment'? If the data
is unavailable, would it be possible for you to make this test for me?

If I can't repeat the analysis directly, can I ask exactly which 'mean
comparison test' was used? i.e. was it the t-test? Can you please
describe in a bit more detail how the regression was done?

Also (sorry!), Figure 1 B/C shows 'standard errors'. Are these
standard errors of the mean? (it looks that way, but I'm just trying
to make sure I have all the details for my 'journal club').

Thanks for getting back to me so promptly, and thanks again for the
interesting work, and for considering the above questions. I'm sorry
that I can't simply work on these questions myself using the
underlying data.

Sincerely,
Dan.



From: "Camelia M. Kuhnen"
To: Dan Bolser
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"
Date: Tue, 13 Oct 2009 20:39:45 -0500
Cc: "jchiao@northwestern.edu"

I am sorry, Dan, but the data set is proprietary and cost us a lot of
money to obtain. We plan to use it in other studies in the future and
at this point we will not make it publicly available. While I
understand there are interesting questions to answer that we have not
addressed in the PLoS paper, such as whether there are gender effects,
I really do not have time right now to do additional analyses to
answer these questions. Please don't take it personally, but I have
other priorities at work at the moment.

Regards,

Camelia



Date: Wed, 14 Oct 2009 03:11:47 +0100
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"
From: Dan Bolser
To: "Camelia M. Kuhnen"
Cc: "jchiao@northwestern.edu"

2009/10/14 Camelia M. Kuhnen :
> I am sorry, Dan, but the data set is proprietary and cost us a lot of money
> to obtain. We plan to use it in other studies in the future and at this
> point we will not make it publicly available. While I understand there are
> interesting questions to answer that we have not addressed in the PLoS
> paper, such as whether there are gender effects, I really do not have time
> right now to do additional analyses to answer these questions. Please don't
> take it personally, but I have other priorities at work at the moment.

No, of course I don't take it personally, I'm just surprised that
there is no way for you to give me the data.

The normal practice in biology is to provide the data under an
institutional 'non disclosure' agreement that basically says that if I
want to publish any findings based on the data that I should get your
express permission to do so first, and abide by any conditions that
you require (usually co-authorship). Failure to abide by the NDA is
then a serious institutional legal issue.

Anyway, could you not provide the ~5,900 'residuals' with only a s/s
yes/no flag and a 7-repeat yes/no flag? I'm struggling to see what I
could do with that data other than simply try to repeat the analysis
that you have presented. I'm assuming this would just be 65 rows of
data with three columns?

Could you or one of your colleagues please let me know how the
difference between the mean residuals in the different groups was
tested for significance? I'm guessing t-test, but I'd like to know for
sure.

Thanks again for your help (and patience).

Sincerely,
Dan.



Date: Wed, 14 Oct 2009 07:09:53 -0500
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"
From: joan.chiao@gmail.com
To: Dan Bolser
Cc: "Camelia M. Kuhnen"

Dear Dr. Bolser,

Thanks very much for your interest in our work and this project more
broadly.

If you would like to collaborate on this project, would you be send us a cv
or some kind of resume so we can better understand your scientific
background and expertise?  I was unable to find a your professional website
online.

Regarding distribution of primary data, I understand the issue of 'open
access' to data, but usually, at least in neuroimaging, this is done by the
authors depositing the data into a repository which then mitigates requests
for data.  As I understand from Cami, in the field of Finance, distribution
of raw data is much rarer, and typically only occurs to collaborators or
known colleagues who work in the same research field.  Finally, as Cami
mentioned earlier, we are still working with the data and have planned a
number of additional analyses and at this time would prefer not to make the
data 'open access'.

In any case, hope that you understand the starting assumptions underlying
our preference not to distribute raw data for the time being.

Sincere regards,
Joan Chiao



Date: Wed, 14 Oct 2009 16:10:04 +0100
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"
From: Dan Bolser
To: jchiao@northwestern.edu
Cc: "Camelia M. Kuhnen"

Hi guys,

Thanks for all your kind, patient and careful replies so far.

First off, to pick up your request, I have put some information about
myself at the end of the email. I must apologise, because I appreciate
that this is a courtesy I often forgo. I hope the information won't
affect your decision about weather or not to answer my email! ;-)

OK, to clarify there are two outstanding issues:
  1) request for the data, and
  2) request for clarification of the methods.

Request for data:

To be clear, I'm not asking for you to make the data open access.

In the first instance, I thought you could 'safely' send me the table
of the 65 "average excess risky investment" scores per subject, along
with a flag for the two genotypes of interest. i.e. 65 rows, three
columns. I want this data so I can repeat the statistical analysis
that you presented in the paper. As a bonus, I thought you could send
me an extra column with the gender of the subjects so I could look for
correlation within those groups. This request still stands, because I
just can't see how releasing that data to me could be in any way
problematic for you.

In the second instance, as Professor Kuhnen raised the issue of
proprietary data, I suggested that I would be willing to sign an NDA,
that would legally guarantee your right to fully control the data and
any results, be they commercial or academic in nature. I am still
willing to sign an NDA for the above data if you see fit to draw one
up.

In case you are worrying, my intention is simply to better understand
the analysis that you published. i.e. I want to look at the standard
deviation of the "average excess risky investment" and also try
various different statistical tests to compare the difference in the
mean, including a comparison of a one-tailed and a two-tailed test.
This is simply for my own curiosity. I do not intend to publish any of
the results, nor do I seek to obtain financial gain from the results.

Request for clarification of the methods:

I asked several questions about the analysis presented in the
publication that I honestly feel are (more or less) reasonable. I feel
that this kind of question is exactly why there is a communicating
author. I'll reiterate my questions here because of the garbled nature
of my previous emails (sorry about that):


1) Figure 1 B and C show 'standard errors'. Are these standard errors
of the mean?

2) Which 'mean comparison test' was used in the comparison? i.e. was
it the t-test?

3) Was there a significant difference between the groups under a
two-tailed significance test?

4) Can you describe in a bit more detail how the regression was done?

5) Is there any significant association with subject gender?

6) Which group of people made the most money during the test? I
suppose this has more to do with the experimental design than the
exact strategy employed.


Please accept my apology if any of these questions are very basic
within your respective fields. As I hinted before, I'm a biologist
(bioinformatician), and I don't think there are such 'standard' tests
in the field.


About me

I'm a bioinformatician working as a post doc research assistant at the
University of Dundee. I have a Ph. D in the bioinformatics of
protein-protein interaction and protein structure analysis. I'm
interested in NGS, genotyping, GWAS, and the future of a
'post-genomic' society. Since you were looking, here are some links:

* http://network.nature.com/people/dan
* http://www.linkedin.com/in/bolser
* http://openwetware.org/wiki/User:Dan_Bolser
* http://www.slideshare.net/danbolser


Please let me know if anything is still unclear.


Sincerely,
Dan Bolser



Date: Wed, 14 Oct 2009 10:40:40 -0500
From: "Camelia M. Kuhnen"
To: Dan Bolser
Cc: jchiao@northwestern.edu
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"

Dan,

I normally do not share data unless requested by the editor of a
journal. However, I will make an exception now just because of the
cultural difference between your field and mine.

I have attached here the data file containing subject-level residual
investment and genotype. This is absolutely the maximum amount of data
that Joan and I will share at this point.

Regarding your questions:

(1), (2) and (3) you can answer on your own based on the attached data
(4) is answerable based on the details in the paper and supplement
(5) and (6) are not the focus of the paper and we will not discuss these
additional results for now. They may be part of other publications.

Good luck with your research,

Camelia



Date: Wed, 14 Oct 2009 17:39:19 +0100
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"
From: Dan Bolser
To: "Camelia M. Kuhnen"
Cc: jchiao@northwestern.edu

Awesome!

Thanks very much for providing the data!


> Regarding your questions:
>
> (1), (2) and (3) you can answer on your own based on the attached data
> (4) is answerable based on the details in the paper and supplement
> (5) and (6) are not the focus of the paper and we will not discuss these
> additional results for now. They may be part of other publications.

With regard to question 2, "Which 'mean comparison test' was used in
the comparison?", I have tried a one-tailed t-test (Welch Two Sample
t-test), a one-tailed ks-test (Two-sample Kolmogorov-Smirnov test) and
a one-tailed wilcox-test (Wilcoxon rank sum test with continuity
correction).

The wilcox-test seems to fit the given p-value for 5-HTTLPR, but not
DRD4 (nor any of the other tests).

Can you enlighten me as to which 'mean comparison test' was used to
obtain the p-value of 0.04?

The lowest p-value I can obtain using the above tests is about 0.07.

Thanks again for providing the data.

Sincerely,
Dan.



Date: Wed, 14 Oct 2009 13:00:38 -0500
From: "Camelia M. Kuhnen"
To: Dan Bolser
Cc: Joan Chiao
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"

See attached pdf -- the one-tailed p-value of the mean comparison test
is 0.0364.

Camelia



Date: Thu, 15 Oct 2009 10:38:41 +0100
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"
From: Dan Bolser
To: "Camelia M. Kuhnen"
Cc: Joan Chiao

2009/10/14 Camelia M. Kuhnen :
> See attached pdf -- the one-tailed p-value of the mean comparison test is
> 0.0364.

Hi Camelia,

Thanks for the information.

I have now resolved the difference that I was seeing between your
results and mine (details included below for your reference).

Would you mind if I wrote this up as a comment for the PLoS One
website? I think this is the kind of thing they would like to see
there. If you agree, I'll send the text for your approval before
posting it there.

Thanks very much for all the help.

All the best,
Dan.


Details:

I found that the problem was that the stats package I use (called 'R')
does not automatically pool the variance of the two groups when
performing the t-test. (Stata clearly does this by default.) The
result is that R uses the Welch (or Satterthwaite) approximation to
the degrees of freedom for when the variances of the two samples is
not equal. Usually the difference in the magnitude of p is not large,
but in this case it was 'important' (0.07 vs. 0.04).

I used Levene's test to check the assumption of equal variance, and
found no evidence for unequal variances (F(1,63) = 0.8758, p =
0.3529).

Changing the t-test to use equal variances now gives me the exact same
result in R as you observe in Stata.



Date: Thu, 15 Oct 2009 13:31:05 -0500
From: "Camelia M. Kuhnen"
To: Dan Bolser
Cc: Joan Chiao
Subject: Re: Request for data "Genetic Determinants of Financial Risk Taking"

Sure, Dan, you have my approval to write up the comment on the PLoS One
website.

Regards,
Camelia

Monday, 10 March 2014

X11 forwarding request failed on channel 0

Simples, I didn't have xauth installed.

Monday, 30 September 2013

Veronica's Adventures








Well edited interactive machinima with a good soundtrack!

Friday, 7 June 2013

Odin Multi Downloader v4.28

This was fiendishly hard to find, so here it is:
https://mega.co.nz/#!JYdUUSqB!PKPMyl0D1jcJn8NPdh13p1qX3SclLLC7Tz16IXp8uA4

The md5sum is: 26a2cf05f8fc5967cc0533d7ae660c71

Use it to flash your i5500!


I put back my stock UK 'Three' image (I5500XWJQ3/I5500H3GJP2/I5500NEJP2) and ops file (EUROPA_v1.0.ops):
https://mega.co.nz/#!RZ8VyICD!CfHWKGV4a1OMnclD6v53nVnqi8M01PKgrWx1GdtPb6w
https://mega.co.nz/#!QBdADKpR!MDh7HH8dNCObRuhdpokfQwaCQ5lEsX8H9Hwjrq4uWk0


The md5sums are: 1ec5924ee453f442f067d545a2c13b46 and d1cfd906d34843e5c98b6cb6a7d6fcc3

Use it to flash your i5500 back to the way the shop gave it to you!

Monday, 11 February 2013

How to Dport proxy to work from home

Because of the gate, a simple -D alone doesn't work. Here is the magic formula:

dmb@percy:~$ ssh -v -o proxycommand="ssh gate proxy %h" \
  dbolser@login -D 23456

and that's that!

Tuesday, 5 February 2013

MySQL DB connection error (solved)



This error was driving me crazy:

1.20.2 installation : DB connection error: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) (localhost).
Check the host, username and password and try again

because everything was correct, and there was nothing in the Apache error_log.

Turns out it was a selinux issue. I was able to connect fine after disabling selinux.