[Talk-cz] Import skript z uir_adr (fwd)
Vlákno 27.8. - 27.8.2008, počet zpráv: 3
On Wed 2008-08-27 01:00:38, Petr Nejedly wrote:
zobrazit citaci
> Pavel Machek napsal(a):
> > Ahoj!
> >
> > je v priloze...
> >
> > Bohuzel tak jak je napsanej zvlada jen asi tak 2 adresy za sekundu
> > :-(. grep ,19800, ho omezuje na jedno PSC, to asi dava smysl vyhodit
> > nebo nahradit Vasim oblibenym PSC.
>
> Nespěchejmež...
Spechat se neda, pocitace jsou pomaly; ta konverze by mela trvat 10+ hodin...
zobrazit citaci
> BH napsal(a):
> > No, nejdriv bych to radsi prozkoumal, zjistil o kolik to nafoukne
> > data, jak je to kvalitni, odlkadil to a pak teprve se rozhodoval
>
> Vzhledem k tomu, ze geotagovanych je zatim jen cca 10% adres, jedna se
> o cca 300.000 nodu, nafouknuti CR o ~20%. Pokud se UIR_ADR pochlapi
> a da to dohromady cele, narostla by nam CR o 200%. Pak bysme dosahli
> paradoxniho stavu vicemene kompletni site silnic prvnich a druhych
> trid a ulicni site, ale bez vetsiny silnic 3. tridy ;-)
Na pochlapeni UIR_ADR bych moc nespolehal.
zobrazit citaci
> > jestli a jak se to tam nacpe. I kdyz si myslim ze ty data by tam byt v
> > OSM mely. Dost to pomuze, jak pri mapovani, tak pri navigaci.
>
> Mely by tam byt urcite. Dulezite je doladit v jakem formatu a hlavne
> nasetupovat proces pro updaty! (Precijen si nechceme zaneradit OSM
> nejakymi 10%, ktere by nam pak vyrazneji komplikovali dodani tech
> zbylych 90%...
Ten ADRESA_KOD by mel pro updaty stacit, ne?
Anyway, tady je dalsi vzorek, mel by byt oznacen podle debaty na
tady, takze pokud jsem neco udelal blbe, reknete...
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
------------- další část ---------------
cat > /tmp/delme.osm << EOF
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.5' generator='JOSM'>
EOF
ID=0
cat adresa.csv | grep -v ",,.$" | grep ,198.., | head -30 | (
while IFS=, read ADRESA_KOD OBJEKT_KOD ULICE_KOD CISOR_HOD CISOR_PIS PSC STAV VZNIK_DNE VZNIK_INFO ZANIK_DNE ZANIK_INFO PCD X Y; do
ID=$[$ID-1]
# echo $ULICE_KOD $CISOR_HOD $OBJEKT_KOD $PSC $X $Y
if [ $STAV -eq 1 ]; then
grep "^$ULICE_KOD" ulice.csv | ( IFS=, read I1 I2 NAME NAME2
grep "^$OBJEKT_KOD" objekt.csv | ( IFS=, read I1 I2 I3 CISDOM_HOD I4
echo "$X $Y" | ~/sf/timetab/cz/geo/jtsk2wgs | ( read LAT LON H
echo $ID $NAME $CISDOM_HOD $CISOR_HOD $CISOR_PIS $PSC $LAT $LON
echo "<node id='$ID' action='modify' visible='true' lat='$LAT' lon='$LON'>" >> /tmp/delme.osm
echo " <tag k='addr:street' v=$NAME></tag>" >> /tmp/delme.osm
echo " <tag k='addr:housenumber' v='$CISOR_HOD$CISOR_PIS'></tag>" >> /tmp/delme.osm
echo " <tag k='addr:red' v='$CISDOM_HOD'></tag>" >> /tmp/delme.osm
echo " <tag k='addr:postcode' v='$PSC'></tag>" >> /tmp/delme.osm
echo " <tag k='uir_adr:ADRESA_KOD' v='$ADRESA_KOD'></tag>" >> /tmp/delme.osm
echo " <tag k='source:addr' v='uir_adr'></tag>" >> /tmp/delme.osm
echo "</node>" >> /tmp/delme.osm
)
)
)
fi
done )
cat >> /tmp/delme.osm << EOF
</osm>
EOF
cat /tmp/delme.osm | cstocs 1250 utf8 > /tmp/delme2.osm
------------- další část ---------------
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.5' generator='JOSM'>
<node id='-1' action='modify' visible='true' lat='50.1046381' lon='14.5605466'>
<tag k='addr:street' v='VlÄkova'></tag>
<tag k='addr:housenumber' v='2'></tag>
<tag k='addr:red' v='1062'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412883'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-2' action='modify' visible='true' lat='50.1047264' lon='14.5607255'>
<tag k='addr:street' v='VlÄkova'></tag>
<tag k='addr:housenumber' v='4'></tag>
<tag k='addr:red' v='1063'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412891'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-3' action='modify' visible='true' lat='50.1048502' lon='14.5608968'>
<tag k='addr:street' v='VlÄkova'></tag>
<tag k='addr:housenumber' v='6'></tag>
<tag k='addr:red' v='1064'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412905'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-4' action='modify' visible='true' lat='50.1049598' lon='14.5611135'>
<tag k='addr:street' v='VlÄkova'></tag>
<tag k='addr:housenumber' v='8'></tag>
<tag k='addr:red' v='1065'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412913'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-5' action='modify' visible='true' lat='50.1050557' lon='14.5612767'>
<tag k='addr:street' v='VlÄkova'></tag>
<tag k='addr:housenumber' v='10'></tag>
<tag k='addr:red' v='1066'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412921'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-6' action='modify' visible='true' lat='50.1054176' lon='14.5616800'>
<tag k='addr:street' v='VlÄkova'></tag>
<tag k='addr:housenumber' v='12'></tag>
<tag k='addr:red' v='1067'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412930'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-7' action='modify' visible='true' lat='50.1056088' lon='14.5615832'>
<tag k='addr:street' v='VlÄkova'></tag>
<tag k='addr:housenumber' v='3'></tag>
<tag k='addr:red' v='1068'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412948'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-8' action='modify' visible='true' lat='50.1054574' lon='14.5613048'>
<tag k='addr:street' v='VlÄkova'></tag>
<tag k='addr:housenumber' v='1'></tag>
<tag k='addr:red' v='1069'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412956'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-9' action='modify' visible='true' lat='50.1053301' lon='14.5607815'>
<tag k='addr:street' v='BratĹĂ VenclĂkĹŻ'></tag>
<tag k='addr:housenumber' v='2'></tag>
<tag k='addr:red' v='1070'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412964'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-10' action='modify' visible='true' lat='50.1054861' lon='14.5605934'>
<tag k='addr:street' v='BratĹĂ VenclĂkĹŻ'></tag>
<tag k='addr:housenumber' v='4'></tag>
<tag k='addr:red' v='1071'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412972'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-11' action='modify' visible='true' lat='50.1056445' lon='14.5604330'>
<tag k='addr:street' v='BratĹĂ VenclĂkĹŻ'></tag>
<tag k='addr:housenumber' v='6'></tag>
<tag k='addr:red' v='1072'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412981'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-12' action='modify' visible='true' lat='50.1058118' lon='14.5602707'>
<tag k='addr:street' v='BratĹĂ VenclĂkĹŻ'></tag>
<tag k='addr:housenumber' v='8'></tag>
<tag k='addr:red' v='1073'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22412999'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-13' action='modify' visible='true' lat='50.1061687' lon='14.5597862'>
<tag k='addr:street' v='BratĹĂ VenclĂkĹŻ'></tag>
<tag k='addr:housenumber' v='10'></tag>
<tag k='addr:red' v='1074'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413006'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-14' action='modify' visible='true' lat='50.1062320' lon='14.5604077'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='2'></tag>
<tag k='addr:red' v='1075'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413014'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-15' action='modify' visible='true' lat='50.1062680' lon='14.5606118'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='4'></tag>
<tag k='addr:red' v='1076'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413022'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-16' action='modify' visible='true' lat='50.1063023' lon='14.5609008'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='6'></tag>
<tag k='addr:red' v='1077'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413031'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-17' action='modify' visible='true' lat='50.1063621' lon='14.5611703'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='8'></tag>
<tag k='addr:red' v='1078'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413049'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-18' action='modify' visible='true' lat='50.1065341' lon='14.5625167'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='10'></tag>
<tag k='addr:red' v='1079'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413057'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-19' action='modify' visible='true' lat='50.1065599' lon='14.5627087'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='12'></tag>
<tag k='addr:red' v='1080'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413065'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-20' action='modify' visible='true' lat='50.1066043' lon='14.5630097'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='14'></tag>
<tag k='addr:red' v='1081'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413073'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-21' action='modify' visible='true' lat='50.1066641' lon='14.5632793'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='16'></tag>
<tag k='addr:red' v='1082'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413081'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-22' action='modify' visible='true' lat='50.1068259' lon='14.5646136'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='18'></tag>
<tag k='addr:red' v='1083'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413090'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-23' action='modify' visible='true' lat='50.1068542' lon='14.5648334'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='20'></tag>
<tag k='addr:red' v='1084'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413103'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-24' action='modify' visible='true' lat='50.1069572' lon='14.5653901'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='24'></tag>
<tag k='addr:red' v='1086'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413111'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-25' action='modify' visible='true' lat='50.1072822' lon='14.5674519'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='26'></tag>
<tag k='addr:red' v='1087'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413120'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-26' action='modify' visible='true' lat='50.1073258' lon='14.5676402'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='28'></tag>
<tag k='addr:red' v='1088'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413138'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-27' action='modify' visible='true' lat='50.1073869' lon='14.5679236'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='30'></tag>
<tag k='addr:red' v='1089'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413146'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-28' action='modify' visible='true' lat='50.1074645' lon='14.5681894'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='32'></tag>
<tag k='addr:red' v='1090'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413154'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-29' action='modify' visible='true' lat='50.1075521' lon='14.5687776'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='34'></tag>
<tag k='addr:red' v='1091'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413162'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
<node id='-30' action='modify' visible='true' lat='50.1074264' lon='14.5690016'>
<tag k='addr:street' v='CĂglerova'></tag>
<tag k='addr:housenumber' v='36'></tag>
<tag k='addr:red' v='1092'></tag>
<tag k='addr:postcode' v='19800'></tag>
<tag k='uir_adr:ADRESA_KOD' v='22413171'></tag>
<tag k='source:addr' v='uir_adr'></tag>
</node>
</osm>
Pavel Machek napsal(a):
zobrazit citaci
> Spechat se neda, pocitace jsou pomaly; ta konverze by mela trvat 10+ hodin...
O to nejde. Jeste jsme se nedomluvili jak to ma vypadat a ty si tu hazis
outer joinama nad CSV v bashi ;-)
Stejne to nakonec nejlepe provede Tomas Kolda (vid ;-)) protoze uz ma v databazi
i ty 3+ roky updatu a u nej ten outer join pobezi asi tak 130ms.
zobrazit citaci
> Na pochlapeni UIR_ADR bych moc nespolehal.
Hmm, pravda, vsechny updaty dohromady daji necelych 24 tisic nove dodanych
souradnic existujicich adres a vseho vsudy 9 (devet) novych adres ktere
maji i souradnice.
Takze z hlediska souradnic jsou relevantni jen updaty 442, 497, 606, 607 a 6
zobrazit citaci
>
>> > jestli a jak se to tam nacpe. I kdyz si myslim ze ty data by tam byt v
>> > OSM mely. Dost to pomuze, jak pri mapovani, tak pri navigaci.
>>
>> Mely by tam byt urcite. Dulezite je doladit v jakem formatu a hlavne
>> nasetupovat proces pro updaty! (Precijen si nechceme zaneradit OSM
>> nejakymi 10%, ktere by nam pak vyrazneji komplikovali dodani tech
>> zbylych 90%...
>
> Ten ADRESA_KOD by mel pro updaty stacit, ne?
Ano
zobrazit citaci
> Anyway, tady je dalsi vzorek, mel by byt oznacen podle debaty na
> tady, takze pokud jsem neco udelal blbe, reknete...
Udelal. Vychazis ze 4 roky starych dat. Viz prvni odstavec.
(Tim nechci nijak krotit tvoji kreativitu, jen ji mirne nasmerovat.
Pokud to mergovani updatu taky napises v Bashi, jsi borec ;-)
Teda ne ze by to neslo...)
--
Petr "Nenik" Nejedly, NetBeans/Sun Microsystems, http://www.netbeans.org
355/113 -- Not the famous irrational number PI, but an incredible simulation!
On Wed 2008-08-27 13:09:41, Petr Nejedly wrote:
zobrazit citaci
> Pavel Machek napsal(a):
> > Spechat se neda, pocitace jsou pomaly; ta konverze by mela trvat 10+ hodin...
>
> O to nejde. Jeste jsme se nedomluvili jak to ma vypadat a ty si tu hazis
> outer joinama nad CSV v bashi ;-)
;-). No, debata se zda-se omezila na 'jak se zapise cislo popisne', z
cehoz usuzuju ze se v podstate shodujem....
zobrazit citaci
> Stejne to nakonec nejlepe provede Tomas Kolda (vid ;-)) protoze uz ma v databazi
> i ty 3+ roky updatu a u nej ten outer join pobezi asi tak 130ms.
No, ono se to stejne bude uploadovat rychlosti 3 zaznamy za sekundu...
;-).
No... kdyby nekde byly updatovany data v CSV, tak nepohrdnu; asi se mi
update opravdu psat nechce.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html« zpět na výpis měsíce