1. Install pYsearch-3.0 somewhere on your PYTHONPATH.
     http://sourceforge.net/projects/pysearch/files/pYsearch/3.0/pYsearch-3.0.tar.gz/download

2. Do Yahoo! image search

     python yahooimagesearch.py holmes "Comet Holmes"

	 --> produces: yimages-holmes.pickle
                   ysiteimages-holmes.pickle

	 (10022 results available)

	 python yahooimagesearch.py hyakutake "Comet Hyakutake"

	 --> produces: yimages-hyakutake.pickle
	               ysiteimages-hyakutake.pickle

	 (6325 results available)

3. Retrieve URLs:

     python geturls.py holmes \
	 | sed s/\'/%27/g \
	 | xargs -t -n 3 -P 10 wget --user-agent="" -q -c --timeout=180 \
	 > get-holmes.log 2> get-holmes.err

	 date
	 # Fri Apr 16 15:40:31 EDT 2010

	 ls holmes-* | wc -l
	 # 2476

     find . -name "holmes-*" -size 0 -delete
	 ls holmes-* | wc -l
	 # 2322

	 # Remove non-JPEGs (the yahoo search is for jpegs; usually HTML results are 404 documents)
	 for x in holmes-????; do (file $x | grep -q JPEG) || (file $x; mv $x $x.deleteme); done
	 rm *.deleteme
	 for x in holmes-????; do mv $x $x.jpg; done

	 ls *.jpg | wc -l
	 # 2309

	 # (mkdir 2010-04-16-holmes)
	 # (mv holmes-* 2010-04-16-holmes/)

solve-field --backend-batch --backend-config 200.cfg --scale-low 10 \
  --no-plots --continue --skip-solved --new-fits none \
  --cancel none --match none --rdls none --corr none --keep-xylist %s.xy \
  --solved-in %.solved --no-tweak holmes-???? > s1.log 2> s1.err
(cancelled after 0045)

for x in 0 1 2; do
  solve-field --backend-batch --backend-config 200.cfg --scale-low 1 \
   --no-plots --continue --skip-solved --new-fits none \
   --cancel none --match none --rdls none --corr none --keep-xylist %s.xy \
   --cpulimit 300 --objs 200 \
   --solved-in %.solved --no-tweak holmes-${x}???.jpg > s${x}.log 2> s${x}.err &
done
for x in 0 1 2; do
  solve-field --backend-batch --backend-config 200.cfg --scale-low 0.05 --scale-high 1 \
   --no-plots --continue --skip-solved --new-fits none \
   --cancel none --match none --rdls none --corr none \
   --cpulimit 300 --objs 200 \
   --solved-in %.solved --no-tweak holmes-${x}???.xy > s${x}b.log 2> s${x}b.err &
done

# 1338 solve

# After putting in uniformization...
G=$(cat good.lst | sed s+-unsolved++ | sed s+.jpg+.xy+)
solve-field --backend-batch --backend-config 200.cfg --scale-low 0.05 \
   --no-plots --continue --skip-solved --new-fits none \
   --cancel none --match none --rdls none --corr none \
   --cpulimit 300 --objs 200 \
   --solved-in %s.solved --no-tweak -v $G >> sc.log 2>> sc.err &

# 1396 solve


(old instructions past here)

3. Extract URLs from search results
     grep -v "^#" y-urls | awk '{print $3}' > yahoo-urls

4. Retrieve URLs
     grep -v "#" y-urls | sed s/\'/%27/g | xargs -t -n 3 -P 10 wget --user-agent="" -q -c --timeout=180

   -> Note that with the '-c', if you do a new yahooimagesearch.py, you should delete the cached yimage-* files.

5. Delete empty files
     find -name "yimage*" -size 0 -delete

6. Extract sources
     ls yimage-???? | xargs -P 7 -n 1 ./extract.sh > extract.log 2> extract-err.log

7. Astrometrically calibrate

backend -c ../backend-710+.cfg -v yimage-00*.axy > backend-0.log 2> backend-0-err.log &
backend -c ../backend-710+.cfg -v yimage-01*.axy > backend-1.log 2> backend-1-err.log &
backend -c ../backend-710+.cfg -v yimage-02*.axy > backend-2.log 2> backend-2-err.log &
backend -c ../backend-710+.cfg -v yimage-03*.axy > backend-3.log 2> backend-3-err.log &
backend -c ../backend-710+.cfg -v yimage-04*.axy > backend-4.log 2> backend-4-err.log &
backend -c ../backend-710+.cfg -v yimage-05*.axy > backend-5.log 2> backend-5-err.log &
backend -c ../backend-710+.cfg -v yimage-06*.axy > backend-6.log 2> backend-6-err.log &
backend -c ../backend-710+.cfg -v yimage-07*.axy > backend-7.log 2> backend-7-err.log &
backend -c ../backend-710+.cfg -v yimage-08*.axy > backend-8.log 2> backend-8-err.log &
backend -c ../backend-710+.cfg -v yimage-{09,10}*.axy > backend-9.log 2> backend-9-err.log &
(on c29)

backend -c ../backend-705-to-709.cfg -v yimage-00*.axy > backend-2-0.log 2> backend-2-0-err.log &
backend -c ../backend-705-to-709.cfg -v yimage-01*.axy > backend-2-1.log 2> backend-2-1-err.log &
backend -c ../backend-705-to-709.cfg -v yimage-02*.axy > backend-2-2.log 2> backend-2-2-err.log &
backend -c ../backend-705-to-709.cfg -v yimage-03*.axy > backend-2-3.log 2> backend-2-3-err.log &
backend -c ../backend-705-to-709.cfg -v yimage-04*.axy > backend-2-4.log 2> backend-2-4-err.log &
backend -c ../backend-705-to-709.cfg -v yimage-05*.axy > backend-2-5.log 2> backend-2-5-err.log &
backend -c ../backend-705-to-709.cfg -v yimage-06*.axy > backend-2-6.log 2> backend-2-6-err.log &
backend -c ../backend-705-to-709.cfg -v yimage-07*.axy > backend-2-7.log 2> backend-2-7-err.log &
backend -c ../backend-705-to-709.cfg -v yimage-08*.axy > backend-2-8.log 2> backend-2-8-err.log &
backend -c ../backend-705-to-709.cfg -v yimage-{09,10}*.axy > backend-2-9.log 2> backend-2-9-err.log &
(on c27)

backend -c ../backend-702-to-704.cfg -v yimage-00*.axy > backend-3-0.log 2> backend-3-0-err.log &
backend -c ../backend-702-to-704.cfg -v yimage-01*.axy > backend-3-1.log 2> backend-3-1-err.log &
backend -c ../backend-702-to-704.cfg -v yimage-02*.axy > backend-3-2.log 2> backend-3-2-err.log &
backend -c ../backend-702-to-704.cfg -v yimage-03*.axy > backend-3-3.log 2> backend-3-3-err.log &
backend -c ../backend-702-to-704.cfg -v yimage-04*.axy > backend-3-4.log 2> backend-3-4-err.log &
backend -c ../backend-702-to-704.cfg -v yimage-05*.axy > backend-3-5.log 2> backend-3-5-err.log &
backend -c ../backend-702-to-704.cfg -v yimage-06*.axy > backend-3-6.log 2> backend-3-6-err.log &
backend -c ../backend-702-to-704.cfg -v yimage-07*.axy > backend-3-7.log 2> backend-3-7-err.log &
backend -c ../backend-702-to-704.cfg -v yimage-08*.axy > backend-3-8.log 2> backend-3-8-err.log &
backend -c ../backend-702-to-704.cfg -v yimage-{09,10}*.axy > backend-3-9.log 2> backend-3-9-err.log &
(on c25)

backend -c ../backend-701.cfg -v yimage-00*.axy > backend-4-0.log 2> backend-4-0-err.log &
backend -c ../backend-701.cfg -v yimage-01*.axy > backend-4-1.log 2> backend-4-1-err.log &
backend -c ../backend-701.cfg -v yimage-02*.axy > backend-4-2.log 2> backend-4-2-err.log &
backend -c ../backend-701.cfg -v yimage-03*.axy > backend-4-3.log 2> backend-4-3-err.log &
backend -c ../backend-701.cfg -v yimage-04*.axy > backend-4-4.log 2> backend-4-4-err.log &
backend -c ../backend-701.cfg -v yimage-05*.axy > backend-4-5.log 2> backend-4-5-err.log &
backend -c ../backend-701.cfg -v yimage-06*.axy > backend-4-6.log 2> backend-4-6-err.log &
backend -c ../backend-701.cfg -v yimage-07*.axy > backend-4-7.log 2> backend-4-7-err.log &
backend -c ../backend-701.cfg -v yimage-08*.axy > backend-4-8.log 2> backend-4-8-err.log &
backend -c ../backend-701.cfg -v yimage-{09,10}*.axy > backend-4-9.log 2> backend-4-9-err.log &
(on c27)

backend -c ../backend-700.cfg -v yimage-00*.axy > backend-5-0.log 2> backend-5-0-err.log &
backend -c ../backend-700.cfg -v yimage-01*.axy > backend-5-1.log 2> backend-5-1-err.log &
backend -c ../backend-700.cfg -v yimage-02*.axy > backend-5-2.log 2> backend-5-2-err.log &
backend -c ../backend-700.cfg -v yimage-03*.axy > backend-5-3.log 2> backend-5-3-err.log &
backend -c ../backend-700.cfg -v yimage-04*.axy > backend-5-4.log 2> backend-5-4-err.log &
backend -c ../backend-700.cfg -v yimage-05*.axy > backend-5-5.log 2> backend-5-5-err.log &
backend -c ../backend-700.cfg -v yimage-06*.axy > backend-5-6.log 2> backend-5-6-err.log &
backend -c ../backend-700.cfg -v yimage-07*.axy > backend-5-7.log 2> backend-5-7-err.log &
backend -c ../backend-700.cfg -v yimage-08*.axy > backend-5-8.log 2> backend-5-8-err.log &
backend -c ../backend-700.cfg -v yimage-{09,10}*.axy > backend-5-9.log 2> backend-5-9-err.log &
(on c29)

At rev 13342.



8. Convert WCS to data points: RA,Dec,radius of the image centers

python datastats.py

(produces file "imagewcs.pickle")

9. MCMC to find the orbits of the comet

python mcmc_comet.py

10. Submit paper






Notes:

-Use median of image times as initial fiducial point for fitting.

-Given orbital parameters, integrate over time the likelihood of the positions in each image.

-There ought to be two parameters: something about how strongly peaked the likelihood is at the center of the image;
second is probability that images don't contain the comet at all.

-We need to write a likelihood function asap!

-At first, use no time information -- then show that we *could* have used it via EXIF.  Perhaps re-run using time.

-forgot "--solved-in" arg; had to do:
   for x in *.axy; do modhead $x ANSOLVIN $(basename $x .axy).solved; done

- Comet orbital elements in Xephem format for PyEphem:
http://www.cfa.harvard.edu/iau/Ephemerides/Comets/Soft03Cmt.txt
it says on 2009-11-23:
17P/Holmes,e,19.1145,326.8536,24.3499,3.620627,0.1430634,0.43287246,139.4847,01/04.0/2010,2000,g 10.0,6.0

Things that we could do better:

-take each search result, extract the site, and hit that site for its first 100/1000 results.



TODO:

-pipe commands to log files
-make short and long machine-readable tables of URLs from y-urls file.






Yahoo jpegs:
http://edge.astrometry.net/job/joblist/?type=sub&subid=test-200902-84587392
-> wcs.tar.gz

HEAT MAP IMAGES:

tar xzf wcs.tar.gz
mkdir -p cache/density
tilerender -x -180 -X 180 -y -85 -Y 85 -w 512 -h 512 -l solid -l grid -l images -A wcsfns -n -q -g 11 -D cache > heatmap.png
tilerender -x 32 -X 80 -y 30 -Y 60 -w 512 -h 512 -l solid -l images -l grid -A wcsfns -n -q -g 4 -D cache > heatmap2.png
cat ephemerides.txt | grep -v "#" | gawk 'BEGIN{print "cairo color 0.3 0.3 1.0 1.0"} {if (FNR==1) {s="moveto";} else {s="lineto";} printf("cairo %s %s %s\n", s, $5 * 15., $6);}' > cairocmds
tilerender -x 32 -X 80 -y 30 -Y 60 -w 512 -h 512 -l solid -l images -l grid -A wcsfns -n -q -g 4 -D cache -l cairo -A cairocmds > heatmap3.png