p.o.d logo

PLANET.OPENSOURCE.DK

Folk:
 Alexander Færøy
 Anders Ossowicki
 Andreas Bach Aaen
 Anton Berezin
 Bryan Østergaard
 Carsten Pedersen
 Christian Jørgensen
 Christian Sejersen
 Christina Rudkjøbing
 Dan Leinir Turthra Jensen
 David Zeuthen
 Erwin Lansing
 Flemming Jacobsen
 Frederik S. Olesen
 Georg Sluyterman
 Henrik Brix Andersen
 Henrik Lund Kramshøj
 Henrik Tudborg
 Jesper Dangaard Brouer
 Jesper Jarlskov
 Jesper Krogh
 Jesper Louis Andersen
 Jesper Nyerup
 Josef Assad
 Kenneth Christiansen
 Kenneth Geisshirt
 Klavs Klavsen
 Kristian Høgsberg
 Kristian Nielsen
 Lars Knudsen
 Lars Sommer
 Lars Sommer
 Leif Lodahl
 Mads Toftum
 Martin Hansen
 Martin Pihl
 Martin Schlander
 Martin von Haller Grønbæk
 Martin von Haller Grønbæk
 Niels Kjøller Hansen
 Nikolaj Hald Nielsen
 Peter Hansteen
 Peter Larsen
 Peter Makholm
 Peter Rude
 Peter Toft
 Phil Regnauld
 Poul-Henning Kamp
 Sune Kloppenborg Jeppesen
 Sune Vuorela
 Søren Bredlund Caspersen
 Søren Hansen
 Søren Hansen
 Søren Sandmann
 Søren Straarup
 Thilo Bangert
 Thomas Alexander Frederiksen
 Thomas H.P. Andersen
 Thor Dekov Buur

Sidst opdateret:
April 19, 2014, 07:00 UTC

Disclaimer:
Dette er folks egne meninger og har ikke nødvendigvis nogen forbindelse til hvad danske opensource-foreninger mener.

Kom på:
Kontakt planet@opensource.dk hvis du føler du vil være her.

Inkluder venligst URL for det feed du ønsker optaget, samt en redegørelse for opfyldelsen af nedenstående krav.

Krav:
  • Du er aktivt involveret i free/open source software (udvikling, foreninger e.l.)
  • Du skriver på skandinavisk og/eller på engelsk
  • Du har en tilknytning til skandinavien
  • Du skriver en gang i mellem om noget relevant for free/open source software-verdenen




Powered by:
Planet
       
April 16, 2014
Full scalability for Netfilter conntracks (April 16, 2014, 11:41 UTC)
My scalability fixes for Netfilter connection tracking have reached Linus'es tree and will appear in kernel release v3.15.

Netfilter’s conntrack have had a bad reputation for being slow. While this was true in the "early-days", it have been offering excellent scalability for established conntracks for a long time now.  Matching against existing conntrack entries is very fast and completely scalable. (The conntrack system actually does lockless RCU (Read-Copy Update) lookups for existing connections).

The conntrack system have had a scalability problem when it comes to creating (or deleting) connections, for a long time now (single central spinlock).  This scalability issue is now fixed.

This work relates to my recent efforts of using conntrack for DDoS protection, as e.g. SYN-floods would hit this "new" connection scalability problem with Netfilter conntracks.

Finally version 3 of the patchset were accepted March 7th 2014 (note Eric Dumazet worked on the first attempts back in May 9th 2013). The most important commit is 93bb0ceb75 "netfilter: conntrack: remove central spinlock nf_conntrack_lock")

Hey, I'm also blogging on the Red Hat Enterprise Linux Blog

I recently did very practical post on Mitigating TCP SYN Flood Attacks with iptables/netfilter, with the hope to provide the world with a practical solution to solve these annoying SYN-flood DDoS attacks, that we have been seeing for the last 20 years.

I've also been touring with a technical talk on the subject, and the most recent version of the slides are here.

Announcing: The IPTV-Analyzer (April 16, 2014, 10:11 UTC)
I'm happy to announce the first official release of the IPTV-Analyzer project, as an Open Source project.



The IPTV-Analyzer is a continuous/real-time tool for analyzing the contents of MPEG2 Transport Stream (TS) packets, which is commonly used for IPTV multicast signals. The main purpose is continuous quality measurement, with a focus on detecting MPEG2 TS/CC packet drops.

The core component is an iptables (Linux) kernel module, named "mpeg2ts". This kernel module performs the real-time Deep Packet Inspection of the MPEG2-TS packets. Its highly performance optimized, written for parallel processing across CPU cores (via RCU locking) and hash tables are used for handling large number of streams. Statistics are exported via the proc filesystem (scalability is achieved via use of the seq_file proc API). It scales to hundreds of IPTV channels, even on small ATOM based CPUs.

Please send bugreports, patches, improvement, comments or insults to: netoptimizer@brouer.com

April 14, 2014
Poul-Henning Kamp a.k.a. phk
Open Source og Penge (April 14, 2014, 08:33 UTC)
Poul-Henning Kamp Hvis det lyder for godt til at være sandt, er det ikke sandt og selvfølgelig er Fri software ikke gratis: Nogen skal betale for pizzaen og de hvide tennissokker i sidste ende. Diverse open source software projekter har "foundations" der prøver at skrabe penge sammen til udvikling og vedligehold...

Leave a comment

Peter Makholm a.k.a. brother
Peter Makholm
Efterhånden kan min tillid til SSL ligge på et meget lille sted. De seneste tre kritiske fejl SSL-implementationerne er tilstrækkeligt til at råbe vagt i gevær. Men dette er ikke den eneste grund til at miste tilliden til SSL. Gennem et styke tid har jeg testet flere websteder med Qualys SSL Lab...
April 13, 2014
Peter Toft a.k.a. pto
Peter Toft I direkte forlængelse af sidste uges blog-indlæg om børneprogrammering satte jeg mig i dag med hjemmesiden http://code.org. Det er et site, som prøver at forøge interessen for at programmere. Man bliver guidet igennem sjove programmeringsøvelser, og rammen er meget kendt: Flappy Bird, Angry Bir...

Leave a comment

Backup i sikker afstand (April 13, 2014, 19:12 UTC)

rackskab i udhus

Rigtige mænd tager ikke backup – de græder. Dette er den klassiske undertitel til en ophængt hardisk med der brudt sammen med smadrede læsehoveder.

Nogle gør som rigtige mænd, andre systemateser det voldsomt. Jeg hører ikke til nogen af yderkategorierne indenfor backup. Jeg har jævnligt taget backup af mine vigtigste data fra een hardisk til en anden. Ikke fast, men en gang i mellem. Egentlig synes jeg det er fornuftigt, at gøre det oftere, men jeg har aldrig fået automatiseret det – lige indtil for nylig. Samtidig har der været andre grunde til, at jeg ikke har være tilfreds med min hidtidige løsning.

  • Hvis der var indbrud – ville tyven så ikke tage begge harddiske med, hvis de står lige ved siden af hinanden?
  • Hvis et lyn rammer ned i installationen, vil alle diske så stå af?
  • Hvis der går brand i huset, vil alle harddiske gå tabt?

Med andre ord, hvilken type fejl var min hidtidige praksis en forsikring imod? Altså backup fra en harddisk til en anden harddisk stående lige ved siden af. Det sikrer helt klart imod klassiske diskcrash, hvor en harddisk bryder sammen. Jeg er ikke vidende om at jeg er ramt endnu. Måske har jeg diske hvor der er opstået checksumfejl på lokale sektorer, men hvor firmwaren har flyttet data til andre områder. Jeg har ikke undersøgt dette og ved ikke om de ret advancerede firmwares der er på harddiske snyder mig. Jeg har ikke haft et datatab af betydning. Den manuelle ustrukturede sikkerhedskopiering, har sikret, at jeg er klar over hvor frisk en kopi jeg har. Jeg går ikke og bilder mig ind at min automatik har klaret ærterne for mig hver nat, mens realiteten er at scriptet er gået i står for måneder siden. Så ved at gå over til automatisk backup, så tilføjer jeg også muligheden for falsk tryghed.

Mit husbyggeri gjorde, at jeg fik installeret min centrale server i teknikrummet inde i huset. Samtidigt beholdte jeg konceptet med et mindre rackskab i udhuset. Dette skab er nu blevet til mit backupskab. Her står en lille nas maskine, som jeg har sat op til at lave automatisk backup af mine vigtigste data. I praksis hjemmekatalogerne. Backup af komplette maskiner har jeg holdt mig fra. Her foretrækker jeg at sætte dem op på ny ud fra en nyinstallation, hvis uheldet er ude. Det er jo bestemt heller ikke sikkert, at jeg vil have noget tilsvarende hardware efter et datatabsuheld.

Til den automatiske backup til mit backup skab har jeg benyttet rdiff-backup og fulgt denne opskrift. Jeg har fået mig et serverrum nummer to i en anden bygning. Ikke en bygning langt fra huset, ikke en bygning koblet på en anden el-installation, men dog placeret så en brand ikke nødvendigvis tager begge serverrum med i købet. Med lidt held tager en fejlstrøm i el-installationen ikke alle harddiske i begge serverrum med sig og sandsynligheden for et indbrud hvor der er brudt ind i både hus og udhus og det allerede noget ældre it-udstyr snuppet begge steder er temmelig lav.

Nu kunne man jo indvende, at det var bedre at tage backup til en lokation meget længere væk. Dette ville kræve en backup løsning hen over internettet og til en lokation, hvor man kan stole trygt på administratoren. At placere alle mine data i skyen er ikke lige mig. Der er for mange eksempler på udbydere der har lukket butikken igen. Der er også langt større sandsynlighed for at virksomheder eller nationer trævler mine data igennem for at lære mig bedre at kende. Det ønsker jeg hverken at give Google eller NSA mulighed for.

Trofaste læsere af denne blog vil kunne genkende racket. Det har været placeret i mit gamle udhus – og nu sidder det så i mit nye – dog med en del mindre udstyr i.

Så backup i sikker afstand for mig er indtil videre omkring 15 meter.

Leave a comment

April 11, 2014
Peter Makholm a.k.a. brother
Peter Makholm
Vi har inden for kort tid set tre meget kritiske fejl i forskellige SSL-implementationer. Kan vi bortforklare det med at SSL er kompliceret og at kryptering er for svært? Desvære nej, det er tankevækkende at alle tre fejl grundlæggende set er dårlig programmering af kode der burde være ligetil at...
April 10, 2014
Poul-Henning Kamp a.k.a. phk
Varnish4 Release-Party d. 29! (April 10, 2014, 20:27 UTC)
Poul-Henning Kamp Vi ved ikke lige præcis hvor det bliver endnu, men vi ved at vi fejrer release 4.0.0 af Varnish et eller andet sted i København d. 29 april. Tilmelding her Party on! phk

Leave a comment

April 08, 2014
Efteruddannelse uden jordforbindelse? (April 08, 2014, 19:04 UTC)

VIA_reklame_set_paa_jpdk_20140408

Nej det er ikke mig der er begyndt at få reklamer på hjemmesiden. Jeg gør en del for at slippe for dem, men på min standard browser på arbejdet, der bliver jeg eksponeret hvis jeg lige checker nyhedsstrømmen, mens compileren snurrer. Jeg faldt over ovenstående reklame. Jeg havde netop skrevet et indlæg på min blog om Schukostik og så kom denne. VIA efteruddannelse har fået lavet et reklamefoto, hvor der bliver slået på metaforen, at man sætter stikket i. I min kontekst kan jeg simplethen ikke lade være med at få øje på, at det er en dansk stikkontakt uden jord, som man forsøger, at sætte et Schukostik i. Det kender vi jo allesammen effekten af: man mangler jordforbindelsen. Det var nok ikke lige den metafor VIA efteruddannelse gik efter. Stikket er desuden også drevet 90 grader i forhold til vanlig praksis.

Leave a comment

April 07, 2014
Poul-Henning Kamp a.k.a. phk
CPU popularitetens forbandelse (April 07, 2014, 21:03 UTC)
Poul-Henning Kamp I dag fylder IBM's S/360 mainframe 50 år. S/360 var den første CPU der blev ramt af popularitetens forbandelse. Kort og godt går forbandelsen ud på at der kommer folk og kaste penge på bordet og siger "Kan I ikke tilføje en instruktion der ..." Mit yndlingseksempel er "MVCOS" instruktionen -- ...

Leave a comment

Schukostik i Danmark (April 07, 2014, 05:00 UTC)

Schuko vægstik

Jeg hører givetvis til blandt de første private huse, der har valgt, at installere Schuko-stik i  huset. Det blev lovligt at bruge disse i dansk byggeri i efteråret 2012.

Men hvorfor dog bruge schukostik? Der var tre gode grunde for mig.

  1. Jeg kunne være med til at bryde LK’s monopol
  2. Mine kontakter vil passe med de schukostik, der sidder på stort set alle apparater. Alt hvad jeg køber kan sættes direkte i og bliver jordet.
  3. Udvalget er større og der er billigere alternatiiver. Konkret kunne jeg købe Cat 6A edb-stik, der var langt billigere end de tilsvarende i LK’s program. Jeg har også højtaler tilslutninger og USB-opladerstik (5V) direkte monteret i samme Eurodåser i væggen.

Da jeg skulle bygge hus, satte jeg blot det som et krav til elektrikeren, at jeg ville have Schukostik fra Gira. Der er en god håndfuld producenter med ret bredde programmer, men jeg valgte Gira, da deres billige standardserie havde et passende enkelt design, der passer med min smag. Elektrikeren tog fint imod kravet. Deres leverandør Solar havde Gira på programmet, så det kunne jo ikke gå helt galt. Det viste sig dog hurtigt, at Solar fik bestilt de forkerte stumper og havde lang leveringstid selv på noget så simpelt som Eurodåser til indmuring. Dårlig produktkendskab og vejledning generelt. De fleste tyske webbutikker kunne have klaret dette bedre. Selv har jeg benyttet bl.a. Voltus, Alles Mit Stecker, Conrad og Reichelt.

De udførende elektrikere på pladsen synes det var fint at prøve at arbejde med andet end LK, men der er ikke noget fedt i at få ødelagt sin planlægning af lange leveringstider på selv simple stumper. Elektrikeren har derfor ikke lyst til at arbejde mere med andet en LK – lige foreløbigt. Så LK’s konkurrenter skal til at tage hænderne op af lommen hvis de vil ind på det danske marked. De skal nok have fat i et typehusfirma, der arbejder med egne elektrikere. Spare nogle basører på materiellet og lidt på den simplere installation. Det er unægteligt nemmere at bore runde huller til Eurodåserne end det er at lave firkantede huller til LK’s dåser. Slutteligt skal slutbrugerne måske præsenteres for nogle af de mere eksklusive muligheder, som LK ikke kan levere. Man kan også få stik med dansk jord – hvis man ikke er villig til at gå hele vejen men blot bruge de europæiske dåser og materiel.

Hvordan fungerer det så i det daglige? Jo først og fremmest, så har jeg skildt mig af med alle vores gamle forlængerledninger. Her er der gode afsætningsmuligheder. Resten af Danmark har ikke konverteret endnu. En del forlængerledninger vr dog så gamle, at de havde det bedst på genbrugspladsen. En kærkommen oprydning. Og så afsted og købe nye forlængerledninger. Her kommer første reelle problem. Man kan ikke købe Schukoforlængerledninger i Danmark. Supermarkederne og byggemarkederne har dem ikke. De har kun forlængerledniger med Dansk jord eller forlængerledninger uden jord. Modellerne uden jord er de værste for de har oftest et stort rundt stikprop, der ikke kan puttes i Schukostikkene. Jeg er altså endt med at måtte købe forlængerledninger i tyskland. Det samme gælder stikpropperne til de få apparater, der rent faktisk har haft dansk jordstik på. Priserne i webbutikkerne på disse dele er enten lavere eller på samme niveau som i Danmark. Dertil kommer porto og typisk 3 dages levering. Dette skal holdes op imod mange byggehandler indenfor få kilometersafstand der har LK materiel på hylderne. Omvendt har jeg kunne købe smarte stikpropper og forlængerledninger/forgreninger, som ikke findes tilsvarende i LK’s program. For et overblik over stik på det danske marked og hvad der passer sammen, så se her.

Alt er jordet. Det først jeg fandt ud af var, at vores elkedel skulle udskiftes. Den slog HPFI-relæet hver gang vi tændte den. vi havde brugt den i mange år, men aldrig sat i en kontakt hvor dens Schukostik blev jordet.Ellers er det virkeligt en fornøjelse, at kontakterne passer til de apparater man køber.

De håndværkere vi har haft efterfølgende har dog haft problemer. De har haft egne forlængerledninger med – med har ikke kunnet sætte det danske jordstik i. De fleste har dog været håndværkere nok til at finde en løsning. typisk en dansk løsning, hvor de ender med ikke at jorde deres udstyr. En tur til byggemarkedet giver dem jo ikke mulighed for at købe et løst Schukostik til at montere på deres kabeltromle. Så jeg skal nok have lavet en lille Schuko til Dansk jordstik konverter. Nogle håndværkere er dog opdaterede. Jeg har været udsat for at maleren lånte elektrikeren en kabeltromle….

 

Leave a comment

April 04, 2014
Ubuntu Global Jam i København (April 04, 2014, 11:04 UTC)

I denne weekend afholder vi Ubuntu Global Jam på Frederiksberg.

ugj09_banner_195x500_yellow_EN

Vi har fået lov til at låne lokaler hos DitLab på CBS og Jammer fra 13-16 både lørdag og søndag.

Der er mulighed for mange forskellige typer aktiviteter fra bug-arbejde og oversættelse, til support og planlægning af kommende aktiviteter. Måske kan vi få startet det praktiske arbejde, så vi kan få en ordentlig 14.04 Release Party i år?
Det er også muligt at komme forbi og se den kommende Ubuntu 14.04 i aktion, eller så support til dit eksisterende Ubuntu system.
Hvilke aktiviteter vi konkret giver os ud i afhænger selvfølgelig af de fremmødte.

Det er på ingen måde et krav at være tilstede hele dagen, det er helt fint at droppe ind i løbet af dagen.

Adressen er
Howitzvej 60, 2000 Frederiksberg

Se Facebook-begivenheden eller LoCo-portal for yderligere info.

April 02, 2014
Peter Makholm a.k.a. brother
Version2.dk: Gør Digital Post til NemMail! (April 02, 2014, 10:23 UTC)
Peter Makholm
Jeg kan godt lide ideen bag NemKonto og NemSMS. Her har det offentlige ikke gået ind og udviklet nogle nye tjenester og trukket dem ned over hovedet på borgeren, istedet har man forsøgt at gøre det let for det offentlige at bruge borgernes eksisterende tjenesteudbydere. Der er sikkert en række fe...
Peter Toft a.k.a. pto
Peter Toft Min datter Louise på snart 11 år har spurgt ind til hvad det er "at programmere". Jeg har nævnt at hun selv kan lære at programmere sin computer. Vi havde endelig god luft i weekendprogrammet sidste lørdag, så Louise og jeg satte os med hendes laptop og åbnede hjemmesiden for Scratch. Med Scratch...

Leave a comment

March 31, 2014
Peter Larsen a.k.a. czar
to yousee or not to yousee wifi hotspot ? (March 31, 2014, 11:40 UTC)
Peter Larsen

Jeg ved ikke om jeg er synsk, jeg ved ikke om det var for nemt at gætte, jeg ved ikke om datatilsynet var det rigtige sted, jeg ved ikke om jeg nogensinde får noget ud af det, jeg ved ikke om det reelt har et formål, men de. 6 marts skrev jeg til datatilsynet fordi jeg i ren afmagt ikke kunne andet, fordi jeg mente det var urimeligt og fordi jeg følte at det her potentielt set kunne være en sikkerhedsrisiko for private / privatlivets fred ud over det sædvanlige hvis det viste sig at være et issue..

d. 21 marts svarede yousee med de havde lukket mit yousee wifi.

Yousee gambler med vores privatliv og vores privat livs fred.

I dag lukkede yousee så for Yousee Wifi nationalt:

http://www.version2.dk/artikel/yousee-fejl-gav-eksterne-brugere-adgang-til-private-filer-57045

Herunder min kopi af email d. 6 Marts stilet til datatilsynet, datatilsynets svar var at jeg kunne få en telefonisk henvendelse og forklaring af deres holdning i 4 uger frem, jeg har siden da gjort datatilsynet opmærksom på det i pressen fremkommende fakta. og forventer at tale med dem i morgen, sikkert med en afvisning, jeg er også i tvivl om datatilsynet er det rigtige forum, men det var i ren afmagt jeg valgte dem som mål for min klage.

- – -
dt@datatilsynet.dk
dato: 6. marts 2014

Yousee har meddelt mig pr email d. 4 marts at man ønsker at bruge min internet forbindelse til et nationalt wifi accesspoint net

Det er frameldt aktivt samme dag (vedlagt) via deres dertilskrevne link.

Dags dato d. 6. marts er wifi’en aktiveret med åben adgang med login igennem mit accesspoint.

jeg gør det gældende at

1) Jeg har aktivt frasagt mig dette

2) Det virker som en urimelig varslingstid og tidsfrist

3) Jeg opbevarer ting på min privatadresse på disklager som er og kan klassificeres som hemmeligt og såfremt at accesspointet er muligt at hacke og dermed tiltvinge sig adgang til mit netværk. Et ikke utænkeligt scenarie.

http://www.theregister.co.uk/2014/01/06/hacker_backdoors_linksys_netgear_cisco_and_other_routers/

4) Det virker som urimeligt at kunder skal dele den sparsomme båndbredte med udefrakommende når yousee i spidsbelasting har endsige svært ved at give kunderne 10% af deres BETALTE hastighed. Dette kan der nemt findes udsagn på på : https://www.facebook.com/youseedanmark?ref=br_tf

Jeg vil derfor gerne bede om datatilsynets udtalelse om følgende:

Er det ret og rimeligt med en frist på 2 dage til optout?

Er det rimeligt at kunder skal lave optout på et sådan tiltag, burde optin ikke være brugt?

Er der en form for sikkerhed eller garanti (økonomisk) som yousee stiller til rådighed såfremt deres teknik fejler og der dermed kan tiltvinges adgang via det ÅBNE accesspoint til hemmelige dokumenter eller forretningshemmeligheder som må findes på kundens private netværk?

Screenshot af wifi scanner, 00:xx:xx:xx:xx:xx er mit private adgang til wifi / 02:xx:xx:xx:xx:xx er yousee wifi-spot ÅBNE adgang.
Begge er mit youseemodem, fysisk placeret i min ejendom, på mit netværk med fysisk adgang til mit interne netværk omend yousee nok vil påråbe sig at det er det softwaremæssigt sikret fra at kunne tilgå mit netværk.

Leave a comment

Poul-Henning Kamp a.k.a. phk
Hvor er YouSees bøde ? (March 31, 2014, 10:14 UTC)
Poul-Henning Kamp Hvis et slagteri sender kontamineret fars på markedet og/eller har problemer med egenkontrollen, kan fødevarestyrelsen idømme en bøde. Hvorfor er der ikke en tilsvarende styrelse der kan idømme YouSee en bøde for ikke at have styr på sikkerheden ? Den bedste måde at forbedre IT-sikkerheden, er ...

Leave a comment

Henrik Tudborg a.k.a. tbug
vpython - easier virtualenv (March 31, 2014, 07:00 UTC)

I - like most python hackers - use virtualenv on a daily basis. I love virtualenv. It allows me to install packages in a local context, without isolating my development environment completely.


What is vpython

Vpython is a tiny(-ish) bash script to help with your day to day virtualenv needs.

  • You don't have to worry about sourcing the activate script.
  • You don't have to point to your virtualenv path.

just use vpython instead of python to invoke your scripts.

How

Vpython works by looking for at virtualenv directory in the directory of the script you invoke. If not found, it will go up one directory and search in the parent. When vpython detects a virtualenv folder, it will use that folder's configuration to run whatever you asked vpython to run, just like had it been python.

$> vpython path/to/project/script.py

Running vpython with first argument being a directory, it will search for a virtualenv directory, just like when invoking it with a script path. When a virtualenv is found, vpython will invoke the virtualenv's python, handing you a python shell for that virtualenv.

$> vpython path/to/project/

If no arguments are given to vpython, it will use you current working directory to search for a virtualenv path.

$> pwd
/home/tbug/src/demo-project
> vpython
Using virtualenv at "/home/tbug/src/demo-project/.virtualenv"
Python 2.7.1+ (r271:86832, Sep 27 2012, 21:12:17)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Use vpython --help to see full help text.

$> vpython --help
Usage:
    vpython --help                                this help
    vpython --pip </path/to/env> [<pip_ags>...]   call the virtualenv pip
    vpython --install </path/to/new/env>          install a new virtualenv
    vpython --find </path/to/search/for/env>      return virtualenv path if found, else exits with non-0
    vpython <python_file>                         call a python file inside a virtualenv
    vpython <directory>                           start a python shell inside a virtualenv

Usecase

This is our folder structure:

/home/tbug/src/demo-project/
|-- .virtualenv
|   +-- { regular virtualenv directory structure here }
+-- script.py

With regular virtualenv, we would do something like

/home/tbug/src/demo-project $> source .virtualenv/bin/activate
/home/tbug/src/demo-project $> python script.py
/home/tbug/src/demo-project $> deactivate

and with vpython

/home/tbug/src/demo-project $> vpython script.py

It might not seem like much to save a line (noone deactivates a virtualenv, right?) and you are right.
The strength of vpython is the ability to look up the virtualenv no matter what path is your current working directory, and even if that script is behind a symlink.

Say our script.py is some utility that we want to use all the time. We might link it somewhere on our path, like /usr/local/bin.

But we still need the virtualenv to activate before the script is run?! What do?!

If we use vpython, the virtualenv configuration will be correctly loaded before running our script.

We can even use vpython in the script's shebang.
Pretend this is our script.py, and that it contains something very useful:

#!vpython
#imaging that we import something useful from our virtualenv site-packages
print "hello vpython world"

Now, if you make that script executable, no matter where you run it from, it will use the correct virtualenv directory. As long as script.py is inside a folder (or subfolder) of a virtualenv directory, it will work.

Let's try symlinking script.py to /usr/local/bin, or wherever you put your stuff:

/home/tbug/src/demo-project $> ln -s `pwd`/script.py /usr/local/bin/script
/home/tbug/src/demo-project $> #it is now liked to our path as "script"
/home/tbug/src/demo-project $> script
Using virtualenv at "/home/tbug/src/demo-project/.virtualenv"
hello vpython world

Note that vpython always tell you where it found the running virtualenv. If you don't want that, you can invoke vpython with the -q or --quiet flag.

Installing vpython

Make sure you have a working python and virtualenv installed (vpython will warn you if they are missing, so don't worry).

  1. Clone the repository from github (https://github.com/tbug/tbug.github.io.git).
  2. Run the installer script (vpython/install.sh).
  3. There is no step 3.

The installer will try to place symlinks to the vpython and vpip scripts in ~/bin if it exists, otherwise it will install it to /usr/bin.

Note that if ~/bin exists, it is assumed that it is also on your $PATH.

Updating vpython

  1. cd to the path of the vpython repo clone.
  2. run git pull.
  3. Once again; no step 3.

Starting a new environment

Vpython also contains wrappers for creating and installing packages inside a virtualenv.

/home/tbug/src/demo-project $> vpython --install .
New python executable in /home/tbug/src/demo-project/.virtualenv/bin/python
Installing setuptools, pip...done.

By default, vpython will install the virtualenv to a folder called .virtualenv. You can change this by setting the ENV_NAME environment variable like this:

/home/tbug/src/demo-project $> ENV_NAME=env vpython --install .
New python executable in /home/tbug/src/demo-project/env/bin/python
Installing setuptools, pip...done.

It is only necessary to set this when installing a new environment. When using an existing environment, vpython doesn't care about the environment name.

Now that you have a new environment, it is time to install some packages.

/home/tbug/src/demo-project $> vpython --pip . install asynckit

The syntax here is:

vpython --pip (virtualenv lookup path) (pip arguments...)

This is a bit cumbersome, so vpython ships with the vpip helper.

vpip assumes your current directory is where you want to look for a virtualenv, so if your current working directory is somewhere within a virtualenv, you can use vpip just as regular pip:

/home/tbug/src/demo-project $> vpip install asynckit
Using virtualenv at "/home/tbug/src/demo-project/.virtualenv"
Downloading/unpacking asynckit
..............................
Successfully installed asynckit
Cleaning up...

Something broke!

Oh dear! If you find an error, open an issue, or send me a pull request with a fix.

March 30, 2014
min mor på nettet (March 30, 2014, 08:47 UTC)

 

www.hannenyboebach.dk

www.hannenyboebach.dk

Det er efterhånden en del år siden, at alle generationer kom på Internettet i Danmark. Det er dog ikke på lige vilkår. Forståelsesrammen er bestemt ikke den samme. Da jeg i 1996 var hotlinemedarbejder hos Danadata (senere TDC Internet), fik vi Danmark på nettet. Jeg mindes en kundetilvækst fra ca. 2000 kunder til over 100.000 kunder på 5 måneder. Det var alle nye kunder. Ikke kunder der var kommet fra andre selskaber. I samme periode fik danske portaler som jubii.dk succes. Stifteren af Danadata Preben Mejer havde også disse tanker. Midt i den hidsige vækst af opkoblinger til nettet (via modem) fløj han i helikopteren hen over og så at portaler ville blive det næste hit. Han fik ret. Meget er dog sket siden. Søgemaskinerne tog over. Først Altavista og derefter Google.

Da min mor kom på nettet havde Google gjort deres indtog. Google blev den foretrukne startside. En overskuelig indgang til nettet. Her for nogle få uger siden gik det op for mig, at hun reelt ikke kunne kende forskel på nettet og Google. Hun havde aldrig indtastet en URL (webadresse) i feltet øverst til venstre i Internet Explorer. Det samme galdt min mors jævnaldrende vininder. De var alle faldet i det som var portalerne store ønske, at gøre deres startside til ikke blot startsiden til nettet, men at gøre siden til selve nettet. Samtidigt med de første store  hamstringer af domænenavne under .dk, så som TDC’s opkøb af alle kommunenavne, da gik snakken i helikopter perspektiv på, at det i fremtiden ikke ville være vigtigt hvilket domænenavn man havde, blot man kunne finde ens hjemmeside via portalerne og søgemaskinerne meget nemt.

Reelt set er tendensen gået begge veje. Ser man på mobiltelefonnumre, så fik jeg i sin tid et forholdsvis nemt huskbart nummer ved at være klar på tasterne, når de nye nummerserier åbnede. Alle numre var lige gode set fra teleselskabernes side. I dag koster det nærmest ekstra, hvis man vil have to cifre efter hinanden der er ens. Der er ekstra pris for bronce, sølv og guldnumre. Omvendt har alle i dag en smartphone med telefonbog i og numre er stort set ikke noget man indtaster længere.

Selv om man ikke forstår internettets tekniske aspekter fuldt ud, så er der alligevel megen glæde ved at være tilstede på nettet. Jeg fik hjulpet min mor med at lave en hjemmeside for hendes billedkunst. En side lavet med WordPress på et betalt webhotel, på et betalt eget domæne. Alt sammen i modsætning til en gratis hjemmeside på en reklamebetalt portal. Altså “Kvaliteten af dit indhold skal sælge vores skrammel” konceptet. Nu har min mor fået en rolig side hvor hendes eget indhold viser hendes eget værd. Dette koster. I dette tilfælde 210 kr om året.Ikke en stor pris for at få foden i eget hus frem for at bo i Bigbrotherhuset. Det kræver selvfølgelig, at man ved om man bor i Bigbrotherhuset. Hvem sagde “The Truman Show” ?

Den nye kunsterside kan findes her.

Leave a comment

March 28, 2014
Arduino to the max: 11x11x11 LED-cube (March 28, 2014, 23:11 UTC)

March 29 2014 is Arduino day, also in Labitat. This is a good opportunity to describe my LED-cube:

 

This LED-cube pulls a number of tricks to get the most out of just a single normal Arduino Uno. A meager 16 MHz and 2048 bytes of RAM goes a long way with sufficient ingenuity and creativity. Here are some highlights:

  • 12-bit PWM, 16 grayscales non-linear.
  • Animations generated on-board, read from SD-card, or streamed over USB.
  • 178 Hz refresh rate, transferring 3 Mbits/s of data to the LED driver shift registers.
  • 50 Hz animation framerate, receiving 269kbit/s of animation data over the serial port.
  • Approximately half of spare CPU time available for on-board generation of animations.
  • Multi-processor Arduino! Uses the Atmega 16U2 USB-handling chip on the Arduino Uno as a second processor, reading animations from an SD-card.
  • Hand-crafted assembler to speed up time-critical parts.

LED-cube basics

One of the first questions I get when I show the LED-cube to people is: How is it possible to turn individual LEDs on and off? After all, it looks like everything is connected to everything. Here is how it works.

The LEDs in the cube are organised into columns. The 11x11x11 cube thus has 11*11=121 columns, each containing 11 LEDs. Let us first see how a single column works:

All the cathodes are tied together and connected to ground. Each anode can be individually connected to the supply voltage using one of the switches. This allows to turn on one LED by connecting the appropriate switch.

Now let us see how it works with multiple columns:

The anodes of one LED from each column are all tied together and connected by a single switch to the supply voltage. The columns now have each a switch connecting them to ground. Now, to turn on eg. D5, we would connect SW2 and SW5. To turn on in addition D8, we would connect in addition SW6.

But what if we want to turn on D1 and D5? For this, we need to connect all of switches SW1, SW2, SW4, and SW5. But that turns on also D2 and D4. Thus, to control every LED independently, we need something more.

The answer is multiplexing. We only ever have one of SW1, SW2, and SW3 turned on at the same time. This allows to individually control each LED in one level of the columns using the bottom column-switches. Then shortly after (half a millisecond in the 11x11x11 LED-cube), we turn off that switch and turn on the switch for the next level, at the same time flipping the column-switches as appropriate for controlling the LEDs at that level. By doing this quickly enough, the human eye is too slow to perceive any flickering, and we get the illusion that each LED is turned on or off independently, at will.

Using multiplexing, the required number of switches and connections is greatly reduced. In the 11x11x11 cube, we need only 11*11=121 switches for the columns, plus an additional 11 swiches, one for the anodes in each horizontal layer. ("Only", compared to 11*11*11=1331). In the cube structure, the column connections are the vertical structures, and the horizontal structures connect the anodes in one layer each to a switch through 11 extra columns at the back of the cube.

Soldering the LED structure requires some soldering skills, but with 1331 LEDs to do, such skill will be naturally aquired before the end of the project. Most especially, patience is needed. There are several good pages on the internet describing in detail how the LEDs are soldered together for an LED-cube, here is one, for example.

The layer/anode switches are implemented using 11 P-channel MOSFETs, controlled directly from 11 GPIO pins on the Arduino. The column switches are implemented using 8 TLC5940 LED driver ICs. The TLC5940 has the additional benefit of being able to accurately control how much current each LED receives, as well as being able to dynamically adjust the LED brightness using 12-bit PWM (4096 intensity levels).

Electronics

The electronics sit on a 20cm-by-20cm PCB located below the base of the LED-cube. The picture shows the PCB from the bottom side; the LED columns (and the connection to the 11 layers) enter through the yellow holes from the top, are bent 90 degrees, and soldered to the copper pads.

The 8 ICs spaced around the middle are the TLC5940 LED drivers, connected to each LED pad. The connections are made for ease of PCB layout, so the software has a lookup table that maps each LED column to the TLC5940 output to which it is connected. Along the right of the board are seen the 11 P-channel MOSFETs that supply the voltage for each layer, one at a time.

At the very right edge is seen the connector to the GPIO pins on the Arduino Uno. Originally, the cube was controlled by an actual Arduino Uno device, connected to the cube with a ribbon cable. But the current version uses instead a custom control PCB with a small, narrow footprint that just fits below the base of the LED-cube. This way all the electronics is hidden away below the very narrow base, resulting in a very nice elegant presentation of the cube:

This control PCB is similar to an Arduino Uno; but it is stripped for anything not needed for the cube, and has a micro-SD card slot and level-shifter soldered directly on the board (originally an small external micro-SD board was connected to the Arduino Uno). The micro-SD slot (not shown on the drawing) and the level-shifter are in the middle. To the left is the USB-connector and the Atmega16U2 chip that handles the USB. To the right is the Atmega328 that runs the main code for the LED-cube. The connections to the main PCB are soldered to the pads along the right and top-right.

The PCBs were layed out with KiCAD; see the end of this article for links to source code. The main board was manufactured with the PCB CNC milling machine we have at Labitat. The small Arduino-clone PCB was manufactured by the SeeedStudio Fusion PCB service.

Software

The LED-cube has 1331 LEDs, each of which can be PWM'ed for 16 different levels of brightness, from 0 (turned off) to 15 (fully on). The state of all the LEDs is stored in a framebuffer; with 4 bits per LED that amounts to 666 bytes. To avoid flicker, double-buffering is needed. With 2kByte of memory, the Arduino has just room for two framebuffers, with a bit of memory to spare for the rest of the activities.

The software has two main tasks to control the LED-cube:

  1. Every 20 milliseconds, load one of the frame buffers with the next frame of the animation being played, for an animation framerate of 50 per second.
  2. Every 500 microseconds, turn on the LEDs in one layer of the cube with the appropriate intensity, by programming the TLC5940 driver chips with the correct PWM value and turning on the power to that layer through the next P-channel MOSFET; resulting in a refresh rate for the entire 11-layer cube of around 178 Hz.
The loading of animation frames into the framebuffers can happen from three sources: They can be loaded from an SD-card, they can be streamed from a PC over the USB port, or they can be generated by code on-board the Arduino itself. The refresh of the LED intensities happens in a timer interrupt, using the SPI device on the Atmega328 to communicate with the TLC5940 driver ICs. The individual steps are detailed in the following sections.

Communicating with the TLC5940 LED-driver ICs.

The TLC5940 LED-drivers need an external PWM clock; this is supplied from the arduino from a timer in PWM mode. An 8 MHz PWM-clock is used; this is the fastest clock that the 16 MHz Atmega328 can generate. One PWM cycle is 4096 PWM clocks, as the TLC5940 uses 12-bit PWM. It is important to always let the TLC5940s run an integral number of PWM cycles; else the brightness of LEDs will be off. So a timer-interrupt runs every 4096 PWM cycles to trigger the TLC5940s; this works out to about 2 kHz for switching each layer on and off, or 178 Hz refresh rate for all 11 layers. The TLC5940 ICs employ a latch, so the timer interrupt first latches new data to get stable PWM timing, then goes to work to send out new data to the driver ICs to be ready to latch at the next timer interrupt.

The 8 TLC5940 ICs have a total of 128 outputs, each of which needs 12 bits of PWM data. That amount to 1536 bits or 192 bytes of data to send to the LED drivers in each refresh cycle. One refresh cycle has a total of 8192 CPU clock cycles, and we need to leave as much as possible of those for generating / downloading the next frame of animation, so this step is carefully optimised for speed.

The data is shifted into the TLC5940s using data and clock pins, so we can use the Atmega328 SPI device in master mode. The maximum speed possible for the SPI device is using an 8 MHz clock, so that means 16 CPU cycles per byte plus a couple extra to load the next byte, as the SPI device is only single-buffered, for a total of around 3500-4000 cycles of the 8192 available.

But we also need to prepare the data to send to the LED drivers. This involves a few steps per TLC5940 output:

  1. Fetch from a lookup-table the index of the LED that is connected to the next TLC5940 output on the PCB (remember, those connections are in more or less random order to optimise the PCB layout).
  2. Use that index into the frame buffer to fetch the 4-bit value that gives the required brightness of the LED.
  3. Use the brightness value to index into another lookup table, which maps each possible brightness into the corresponding 12-bit PWM value. This mapping is made non-linear to improve the dynamic range available with just 16 different levels, as the lower PWM values are perceived by the human eye to be much easier to distinguish than the higer values.
  4. Pack the 12-bit values of two consecutive outputs into three bytes suitable for transmission on the SPI device (it is thus easier to do two LEDs at a time, producing three whole bytes of SPI data for each loop iteration).
As it turns out, these operations together take something like 60 CPU cycles for two LEDs or 20 cycles per SPI output byte, which is another 3500-4000 cycles.

With 8192 cycles total, we cannot afford to first spend 4000 cycles preparing the data, then spend another 4000 cycles sending the data out. So we use a trick. By writing the code in assembler and carefully timing each instruction, we can overlap the two operations, inserting into the data generation algorithm instructions to feed the next byte of data to the SPI at exactly the points in time where they are needed.

In C it might look something like this, with the data generation algorithm on the left and the SPI data output on the right:

    // Loop over the TLC5940 outputs two at a time
    for (o = 127; o >= 0; o = o - 2)
    {
      // Find index into frame buffer
      led_idx = led_map[o];                                           // Output one byte
                                                                      SPDR = byte1
      // Load the 4-bit intensity from the frame buffer
      if (led_idx & 1)
        intensity = frame_buffer[(offset + led_idx)/2] & 0xf;
      else
        intensity = frame_buffer[(offset + led_idx)/2] >> 4;

      // Loopup the 12-bit PWM value from the intensity.
      pwm1 = pwm_lookup[intensity];
                                                                      // Output one byte
      // Same for second output                                       SPDR = byte2
      led_idx = led_map[o+1];
      if (led_idx & 1)
        intensity = frame_buffer[(offset + led_idx)/2] & 0xf;
      else
        intensity = frame_buffer[(offset + led_idx)/2] >> 4;
      pwm2 = pwm_lookup[intensity];

      // Pack the two 12-bit PWM values into three SPI bytes
      byte1 = pwm1 >> 4;
      byte2 = (pwm1 & 0xf) << 4 | (pwm2 >> 8);                        // Output one byte
      byte3 = pwm2 & 0xff;                                            SPDR = byte3
However, the algorithm is not really expressible in C. The instructions to output the data bytes to the SPI device need to be placed so that the bytes to output have been computed and the timing between them must be just right so that they happen as soon as the SPI has completed the last transfer (but no sooner, or the previous byte will be corrupted). The actual assembler code can be seen here, in the function shift_out_frame_spi(). The comments in the function mark the cycles spent on each instruction so that the SPI output can be spaced out with the correct timing.

This way, the code is able to send out data for two TLC5940 outputs in around 63 CPU cycles, or about 4000 cycles total per refresh cycle, leaving half the CPU time free for handling the frames of animation, which is nice. I think this is a rather interesting programming technique. It is a bit like multi-threading, but with the instruction scheduling hardcoded explicitly into the program.

Serial reception

In addition to generating some animations on the Arduino itself, they can be streamed into the Atmega328 through the serial port. The protocol is mostly the raw binary data in the framebuffer (4 bits per LED), plus a couple of control bytes like start/end marker, frame number, and checksum, to facilitate synchronisation between sender and receiver. If the receiver detects that the frames are not received correctly, it sends back an error byte; the sender notices this and pauses the data for a few frames, and the two ends re-synchronise. This is a simple but effective technique that allows for efficient transfer of the data.

One frame of data on the serial port is 672 bytes inclusive control bytes. At 50 frames of animation per second, that amounts to 33600 bytes or 268800 bits per second. The serial port is run at 500000 bits per second; with the overhead of start and stop bits, that still allows to comfortably transfer the required data at the desired rate.

However, with this rather high data rate, care is needed to be able to process all received bytes sufficiently fast that no data is lost. The Atmega328 has just a single byte receive buffer. At 500kbps, a new byte arrives 50000 times per second, meaning that we have just 320 CPU cycles to process a received byte before it will be lost due to being overwritten by the next byte.

To handle this, a serial receive interrupt is employed. The interrupt is triggered whenever a byte is received by the serial device on the Atmega328, and we need to ensure that it will be serviced within at most 320 CPU cycles. The Atmega328 does not have interrupt priorities, but it does support nested interrupts. Interrupts are automatically disabled whenever an interrupt routine is invoked, but that routine can re-enable interrupts explicitly, and this will allow another nested interrupt to be handled before the first one is completed. Indeed, this is absolute necessary to do in the cube in the refresh timer interrupt, as this runs for several thousand cycles. Nested interrupts work well, but they require a lot of care; race conditions between conflicting interrupts can be quite hard to debug, and one also needs to protect against runaway interrupts (where the same interrupt is invoked recursively and repeatedly on top of itself until the stack is overrun).

With more than 30000 serial interrupts per second, we also want to make the code for the serial interrupt handler as efficient as possible. Unfortunately the AVR architecture does not exactly shine in this respect. Here is how a typical interrupt routine looks as generated by GCC:

    push    r1
    push    r0
    in      r0, 0x3f
    push    r0
    eor     r1, r1
    push    r16
    push    r17
    push    r18
    push    r19
    push    r20
    push    r21
    push    r22
    push    r23
    push    r24
    push    r25
    push    r26
    push    r27
    push    r28
    push    r30
    push    r31

    ...

    pop     r31
    pop     r30
    pop     r28
    pop     r27
    pop     r26
    pop     r25
    pop     r24
    pop     r23
    pop     r22
    pop     r21
    pop     r20
    pop     r19
    pop     r18
    pop     r17
    pop     r16
    pop     r0
    out     0x3f, r0
    pop     r0
    pop     r1
    reti
That is no less than 40 instructions just as pre/post-ample, most of which take two CPU cycles each.

Of course, in an interrupt routine, we do need to save/restore all registers used. However, most of the invocations of the serial interrupt do not need to use more than a few registers; just enough to grab the next byte from the serial device and put it into the frame buffer. Only for the control bytes at the start and end of a frame do we need more registers for more complex processing. Unfortunately, GCC always generates the code to push and pop all the registers unconditionally, even though some of them are only used in rarely executed code paths (the large number of callee-save registers in the AVR calling convention plays a part of the problem here).

The solution is to write the serial interrupt in hand-optimised assembler. In the fast path, where we are just stuffing a byte into the framebuffer (and computing a checksum on-the-fly, incidentally), we only need to save three registers (plus the condition codes). That all can be done in just 26 instructions. Then in the slow path, the assembler code goes on to push all remaining registers and defer to the more complex processing in a C function.

The actual code can be seen here. The assembler code for the fath path is in serial_interrupt_rx_naked(), while the slow path is in the function serial_interrupt_slow_part().

Reading animations off SD-card

While some of the animations can be computed on-the-fly on-board the Arduino, some of the more complex ones are too much for the puny Atmega328 to handle. The problem usually is not so much processing speed as memory: With a total of just 2048 bytes of RAM, most of which is already reserved for frame buffers and stack space and various global variables, not a lot is left to keep track of stuff like position and velocity of lots of particles in the fireworks animation or similar stuff. Using the serial port, we can generate the animations on a PC and stream them to the cube; however it is also nice to be able to run the cube completely standalone: just plug it into power (or even run it off a battery) and it runs and displays animations on its own, without needing a laptop on tow. Thus the idea was born to pre-compute the animations and read them from an SD-card.

Now, at just 672 bytes per frame of animation, a 4 GB SD-card can store more than one day worth of animation, so this is fine. The problem however is that we are running short on IO as well as on memory. Mostly all available pins are already needed to control the 11 MOSFETs and to communicate with the TLC5940 ICs. Besides, the SPI device, which is normally used to communicate with an SD-card, is needed for communication with the TLC5940s, and the serial device (which can also do SPI) is needed for the USB-to-serial interface. So what can be done?

Let us take a closer look at the Arduino Uno, the top left corner near the USB connector:

What we see here is the Atmega16U2 IC, which comes pre-installed with firmware (based on LUFA to control the USB port and function as USB-to-serial proxy. It is however perfectly possible to modify that firmware to do additional things - such as connect an SD-card! Furthermore, just next to it we have the ICP header - this is normally used to program the flash on the Atmega16U2, but it has the pins for the SPI device, which is just what we need to communicate with an SD-card over SPI. And finally, we even have unsoldered pads with four spare GPIO, one of which we will need as a chip select pin for the SD-card.

So as it turns out, the Arduino Uno is in fact a multi-processor device! The LED-cube exploits this, connecting the SD-card to the ICP pins and spare GPIO of the Atmega16U2 MCU and hacking the USB firmware to handle the SD-card. If there is no activity on the USB port, the hacked firmware will check for the presence of an SD-card with animation data on it; if found, data will be streamed from the SD-card over the serial port to the Atmega328, which will handle the serial data the same, whether it originates from a PC at the other end of the USB, or from the SD card.

Now, using the Atmega16U2 in this way does present some challenges. The Atmega16U2 is only equipped with a meager 512 bytes of RAM, some of which is already needed for LUFA data and so on. The data on SD-cards is read one sector at a time, and a single sector is 512 bytes, already more than the RAM we have left. Most libraries for reading SD-cards and dealing with the FAT filesystem on them is based on reading one sector at a time into a buffer in RAM and processing it there; that just will not work when we have only a few hundred bytes of RAM to spare for the task.

Furthermore, most SD-card/FAT libraries are written in a traditional blocking style. That means, they provide some function you can call to read data from a file on the SD-card. Such function will take a memory buffer (which we do not have the memory for), and it will not return to the caller until all of the requested data has been read, which means waiting at least for one sector to be read. That does not integrate well with the existing USB/LUFA firmware, which runs its own main loop that waits for activity on the USB device and does not return to the main program unless there is some activity to respond to.

To overcome these challenges, I wrote a small event-driven FAT library, seen in ev_fat.h and ev_fat.c. This library works in a streaming fashion, without any blocking. It never needs to process SD-card data in a memory buffer. Instead, the caller feeds it the bytes read off the SD-card one by one, and the library processes the bytes as they are received, keeping track of its state in a small data structure, and returning status information back to the caller about which sectors from the SD-card to read next.

    /*
      Open a named file in root dir of FAT file system.
      Before calling, st->state must be initialised to 0.
      Then the function must be repeatedly called until it returns
      EV_FILE_ST_DONE or negative error code EV_FILE_ST_E*.

      The returned status tells the next action to take, see comments in struct
      ev_file_status for details.

      When EV_FILE_ST_DONE is returned, the first sector of the file, and the
      length in bytes of the file, is returned in st->st_get_block_done.
    */
    int
    ev_file_get_first_block(const char *filename, struct ev_file_status *st);

    /*
      After opening a file, this finds the next sector in the file. When calling
      this function, st->st_get_block_done must be set to / retain the value set
      by the previous call to ev_file_get_first_block() /
      ev_file_get_next_block().  After EV_FILE_ST_DONE is returned the new sector
      number is then found in st->st_get_block_done.
    */
    int
    ev_file_get_next_block(struct ev_file_status *st);

    /*
      This callback is used to stream bytes read as a response to a request
      EV_FILE_ST_STREAM_BYTES. Each byte requested must be passed in, in
      sequence. The return value is true if no more data needs to be streamed;
      in this case it is permissible, but not required, to stop the read early
      and not stream the rest of the requested bytes.
    */
    int
    ev_file_stream_bytes(uint8_t byte_read, struct ev_file_status *st);

With this library, the reading of the SD-card can be handled completely inside an SPI interrupt routine, without disturbing the LUFA USB code. Each time a byte has been processed in the communication between the Atmega16U2 and the SD-card, the SPI device triggers the SPI interrupt. This interrupt processes any byte received, updates its internal state, and loads the next byte to be processed into the SPI device data register. The interrupt is seen in ISR(SPI_STC_vect). The code handles the protocol to connect to and initialise the SD-card, and then takes care of reading in sectors and passing the bytes to the event-driven FAT library.

When we get to actually read real file data out of the SD-card, we stream it directly to the serial port (where it will be received and processed by the Atmega328), to avoid the need for large memory buffers. The existing firmware already has a small FIFO used to buffer data for sending down the serial line. We re-use that, so that when no data is available from the USB for a few seconds we start filling up the FIFO with data from the SD-card instead. A serial device interrupt is triggered whenever the previous byte has been fully transmitted down the serial line; this interrupt fetches the next byte from the FIFO and loads it into the serial transmit data register. If the SD-card delivers data faster than the 500kbps serial line can transmit, we temporarily pause the SPI communication and resume it once the serial interrupt has made room for more data in the FIFO; the SD-card specifications explicitly mention this as a supported way to operate, precisely to help very small microcontrollers be able to process data without requiring excess buffering capabilities.

The end result is an extended USB firmware that retains all the original functionality (streaming serial data from a PC and so on; even flashing the Atmega328 over the serial port still works). And in addition, if the USB is idle and an SD-card is present, data is instead continuously streamed from the card, allowing completely stand-alone operation of the cube.

The code to handle all this does end up rather intricate, as can be imagined. Apart from the need to write a custom FAT-reading library, the precise timing between the different interrupt handlers end up requiring quite a lot of careful coding and debugging. But in the end, I found that code to be quite an interresting exercise, and fun as well - and this is after all a for-the-fun-of-it type project.

Calculating the animations

One of the nice thouches of the visuals this LED-cube in particular is the availability of 16 different intensity levels. This allows for some nice effects, such as fading the LEDs in-out to give a warmer, light-bulb-like perception, and using anti-aliasing to greatly reduce the disadvantage of the very limited 11-by-11-by-11 resolution.

All the animations are computed by this C++ program. The code is mostly a lot of math using vector computations, trigonometry, random number distributions, permutations, physics simulations and other nice stuff. The end result is a sequential stream of animation frames that can be send directly to the LED-cube over the serial port, or stored in a file on an SD-card for stand-alone playback.

Conclusions, and source code

If I were to do this project today, I would probably use an ARM microcontroller like the STM32F4. Such a microcontroller is easily able to handle driving something like this LED-cube without the need for any special tricks due to its much larger memory and performance. But this was just a for-fun project, and it was interesting to see just how much could be squeezed out of the very popular AVR-based Arduino. That is quite a lot, as it turns out.

The nice thing about the LED-cube is: On the one hand it involves lots of tricky programming and advanced technology. On the other hand it has an immediate appeal to many different kinds of people, as is seen whenever we take it on display and it immediately draws the eyes of people passing by. The technology aspect is much harder to appreciate than the visual aspect. I have wanted to write up this article describing the project for some time, in all the gory technical details. I hope a few people will be able to use this write-up to appreciate the technical side as well as the visual side.

All of the code and design files for this project are available on Github under an open source license (GPL):

In addition, the Arduino code needs these header files by Esmil for easy access to Atmega registers and so on.

Poul-Henning Kamp a.k.a. phk
Softwarepatenter, Pop-Corn, Mandag (March 28, 2014, 22:43 UTC)
Poul-Henning Kamp Det er på mandag USAs højesteret hører sagen "Alice Corp" om softwarepatenter. Jeg har lige skrevet et blogindlæg om det ovre på ing.dk og på mandag aften når jeg har læst "transcript" kommer der endnu et indlæg om hvad der skete også derovre. Dommen falder inden Juli. phk

Leave a comment

March 26, 2014
Peter Toft a.k.a. pto
Facebook på mobilen: SKRID! (March 26, 2014, 21:12 UTC)
Peter Toft Vi kender det nok alle sammen. Ny version af en app på mobilen. Den kræver godkendelse, hrrm - bla bla ... Jeg vælger "Opgrader". Jeg tror vi alle ved, at nogle af de ting vi siger ja til er "for meget", men Facebook på Android er lige kommet ti kilometer ud over hvad jeg synes er ok. Læs list...

Leave a comment

March 24, 2014
Martin Schlander a.k.a. cb400f
Jolla Smartphone Part 2 (March 24, 2014, 17:34 UTC)
Martin Schlander

Two months ago I gave an initial review of the Jolla smartphone after one week of use. This is just quick update to the original post.

A lot has happened in the meantime, two major updates have been released and SailfishOS is now officially out of beta. These updates have fixed many issues such as the power drain bug and added many new features including landscape support to the web browser and messaging app. MMS support has been promised for next month’s update. By then Jolla should have all the basic functionality covered.

There has been a steady trickle of nice new native apps to the Jolla Market, including some familiar ones ported from the Nokia N9. And I have also made new discoveries in the Android app world, among others the F-Droid appstore, offering free/open source applications only.

In my original post I claimed that the NavFree navigation app for Android worked well on the Jolla, but as it turns out I hadn’t tested it sufficiently. Instead I have found that the Android apps OsmAnd and MapFactor: GPS Navigation work quite well for gratis, offline, voice GPS Navigation.

20140324172932

I still couldn’t be happier with my Jolla.


March 23, 2014

tidlig_husskitse

Når jeg skal folde mig ud kreativt, så tager jeg ofte papir og blyant i brug. Det giver for mig en hutigere skitse end at kaste mig direkte over et tegneprogram til computeren. Vi havde fået ideen om et nyt hus på vores grund. Grunden er skrående mod syd og lokalplanen stiller en hel række krav til byggeriet. Huset skal være i blanke mursten. Huset skal flugte vejen i en bestemt afstand, taget skal være et saddeltag med en hældning mellem 25 og 35 grader. Det skal være et tegltag. Der må højest bygges 7,5 meter i højden. Der skal være en parkeringsplads foran en eventuel carport. Bebyggelsesprocenten er på 30%.

Huset skulle have flere kvadratmeter end det gamle og samtidigt ikke ligge beslag på hele grunden.

Nu kunne man tro, at så mange restriktioner er en hindring for den frie kreativitet, men jeg vil nu mene, at netop ydre krav er med til at presse en til at komme på kreative tanker. Jeg overførte min ide om en tilbygning i forskudte plan til hele huset. Det kunne hermed lade sig gøre, at lave et hus, der udnytter den skrånende grund, holder sig under 7,5 meter over niveau, er mere kubisk – og derfor får mindre overflade i forhold til gulvplan og dermed holder bedre på varmen. Det gav mulighed for naturligt at få et stort glasparti mod syd, så vintersolen kan udnyttes, blot skal der et stort udhæng til, så sommersolen ikke varmer huset for meget op. Huset er omkring 9 meter dybt for at opnå fornuftige værelsesdimentioner på alle etager. En af mine grundideer var, at et værelse på 3×3 meter er bedre end et værelse på 2×5 meter. Det er ganske simpelt nemmere at indrette, med de typiske størrelser senge og borde nu har. Det meget dybe hus førte til et meget kort hus. Det kortere hus gav mulighed for en bredere carport, så der kan blive plads til både bil og cykler.under tag.

 

husgavl

Den tidlige skitse blev stort set til virkelighed. Jeg havde ikke regnet rigtigt mht. hvor meget en etageadskillelse fylder. De 7,5 meter er hvor meget man må bygge over det eksisterende plan, så den nedgravede del af kælderen tæller ikke med i højden. Taghældningen blev på de minimale 25 grader i henhold til lokalplanen. Det kreative var at, kravet om saddeltag ikke havde specificeret at de to ben skulle være lige lange. Netop det gjorde, det muligt at lave to etager til den ene side og en etage til den anden side og en glidende overgang mellem de to øverste plan. Det korte ben af taget peger mod syd og er derfor velegnet til solceller.

Vores hus ligger i et kvarter med mange krav i lokalplanen. Sådan ser nye lokalplaner typisk ikke i ud. I alt fald ikke i Århus. Her er man som regel kun begrænset af en bebyggelsesprocent og en maks højde på typisk 8,5 meter. Det er også normalt der sættes energikrav på f.eks. 2015-klasse byggeri, som defineret i BR2010. De manglende begrænsninger giver nogle andre boligområder. Der bliver mange forskellige huse. Pudsede hvide bungalover, et plans murstens parcelhuse, træhuse, betonteglstage og paptage. Alt sammen blandet sammen i een pærevælling. For mig at se, så er rigtig mange af de gamle og attraktive kvarterer netop kendetegnede ved, at de har en vis form for ensartethed i arkitekturen. Men mon ikke der vil gå en del år endnu inden man igen tør sætte flere krav formsproget i de enkelte nyudstykninger. De helt frie tøjler der nu er, tror jeg ikke er fremmende for den kreative tankegang, der kan sikre spændende byggeri, der vil vedblive at være attraktivt at bo i.

Leave a comment

March 22, 2014
Peter Toft a.k.a. pto
Peter Toft Jeg har kodet en del i Python de sidste mange år samt C/C++. Jeg fik dog et spørgsmål på bordet den anden dag, som er af generel interesse - hvordan debugger man effektivt efter hukommelsesproblemer i Python på en Linux-maskine? Hvis man f.eks. ser at et Python pludselig er gået fra 100 MB til 5 ...

Leave a comment

March 20, 2014
Poul-Henning Kamp a.k.a. phk
Fest hos mekanikken (March 20, 2014, 09:56 UTC)
Poul-Henning Kamp Der er næppe nogen mere socialt skræmmende oplevelse for en ung mand, end at blive slæbt med til den første store familiefest hos svigermekanikken. Hvordan hilser man hos den klan ? Er det faste håndtryk, high-five eller krammere ? Hvor mange underlige onkler og svogre er der som man bare skal...

Leave a comment

March 19, 2014
Peter Toft a.k.a. pto
Peter Toft Jeg er super glad. Der er 1678 besvarelser på mit spørgeskema om brugen af Linux/BSD fra forrige blog-indlæg - TAK! Det er interessant læsning. Hvilken Linux/BSD distribution anvender du? I har svaret Ubuntu 526 31% Debian 223 13% Linux Mint 173 10% Arch Linux 102 6% Centos 9...

Leave a comment

March 17, 2014
Poul-Henning Kamp a.k.a. phk
En Svær Opgave (March 17, 2014, 10:59 UTC)
Poul-Henning Kamp Nogle gange når folk inviterer mig til at holde et foredrag er der fri skydning i det muntre køkken, andre gange får jeg stukket et mere eller mindre detaljeret oplæg ud. På Onsdag skal jeg holde et foredrag på et temmelig detaljeret oplæg, til Dansk IT's store "Fætter/Kusinefest" i Musikhuset i...

Leave a comment

March 16, 2014
Peter Toft a.k.a. pto
Peter Toft Jeg lavede for nogle uger siden en undersøgelse i min blog over hvilken revisionskontrolsystemer, der bliver anvendt pt. Resultaterne var interessant læsning. Jeg vil gerne gentage succesen men denne gang sætte focus på hvilke Linux/BSD-distributioner, der bliver anvendt i Danmark. For få år sid...

Leave a comment

March 14, 2014
Tivoli Fredagsrock 2014 (March 14, 2014, 12:03 UTC)

Fra alle mig til alle jer…

Tivoli Fredagsrock kalender som Google Calendar. Om Fredagsrock egentlig burde omdøbes til FredagsPop vil jeg slet ikke forholde mig til… under alle omsætndigheder, velbekomme.

Bemærk at der to dage i april afholdes Offspring, en to dages musikfestival i Tivoli.

Direkte links til: ical .ics, html og xml

3153753180_5202de1253

Photo by: Stig Nygaard

Parallel replication is in MariaDB 10.0. I did some benchmarks on the code in 10.0.9. The results are quite good! Here is a graph that shows a 10-times improvement when enabling parallel replication:

The graph shows the transaction per second as a function of number of slave worker threads, when the slave is executing events from the master at full speed. The master binlog was generated with sysbench oltp.lua. When the binlog is enabled on the slave and made crash-safe (--sync-binlog=1 --innodb-flush-log-at-trx-commit=1), the slave is about ten times faster at 12 worker threads and above compared to the old single-threaded replication.

These results are for in-order parallel replication. With in-order, transactions are committed on the slave in strictly the same order as on the master, so that applications do not see any differences from using parallel replication. So no changes to the application are needed to use parallel replication; this is just standard sysbench 0.5 with a single table. This makes parallel replication particularly interesting, as it can be used with any existing applications, without the need to eg. split the data into independent schemas as is necessary with out-of-order techniques like the multi-threaded slave feature in MySQL 5.6. It does however make it much harder for the slave to find parallelism to exploit in the events received from the master, which is why it is interesting to see how much improvement can be obtained from normal workloads.

(MariaDB 10.0 does also support out-of-order parallel replication, but that will be the subject of a different article).

The hardware used for the sysbench oltp.lua is the same machine I used to benchmark group commit previously; I am told this is a machine that is typical for a "standard" datacenter server, with decent I/O on a RAID controller with battery-backed-up cache. Sysbench was run with 10 million rows in one table. The mysqld was configured with 16GB buffer pool and 2 times 1.9 gigabyte redo logs. The different graphs are as follows:

  • binlog, crash-safe: --log-slave-updates --sync-binlog=1 --innodb-flush-log-at-trx-commit=1
  • no binlog, durable: --skip-log-slave-updates --innodb-flush-log-at-trx-commit=1
  • no binlog, non-durable: --skip-log-bin --innodb-flush-log-at-trx-commit=2
  • binlog, non-crash-safe: --log-slave-updates --sync-binlog=0 --innodb-flush-log-at-trx-commit=0

For this test, the master was configured with --binlog-commit-wait-count=12 --binlog-commit-wait-usec=10000. This allows the master to delay a transaction up to 10 milliseconds in order to find up to 12 transactions that can commit in parallel; this helps a lot in improving parallel replication performance, since transactions that commit in parallel on the master can be executed in parallel on the slave.

Adding such delay will be acceptable for many applications to speed up the slaves; in fact in my test it did not affect master throughput at all. One attractive option might be to set up monitoring of the slaves, and if they start falling behind, then the commit delay on the master can be temporarily increased, throttling the master a bit while allowing slaves better possibility to catch up.

The other source of parallelism on the slave is that irrespectively of how the transactions were executed on the master, the commit steps of different transactions can always be applied in parallel on the slave. This is particularly effective at improving performance when the commit step is expensive, as happens when a durable, crash-safe configuration is used. This is seen in the benchmark, where the speedup is particularly large when the slave is configured to be crash-safe and durable, to the point where parallel replication almost eliminates any performance penalty for enabling crash-safe binlog on the slaves. But significant speedup is seen in all the configurations.

(In fact, if you look closely, you will see that turning off the binlog ends up decreasing the performance of the slave. This is bug MDEV-5802, and performance should improve when binlog is disabled when this bug is fixed).

I think these are very promising results. I hope this will inspire users to give the new feature a test on real workloads, and share their experiences.

Exploring the limits of parallel replication

I also wanted to see how the code works for workloads that are not favorable to parallel replication. For this I use sysbench update_index.lua. This benchmark creates transactions with a single primary-key update. With just 10 million rows in the table, the test is in-memory, and the actual work spent for the single-row update is rather small compared to the overhead of reading and parsing binlog events and scheduling the work between the various replication threads. So it is interesting to see if parallel replication can still give some smaller benefit here, or at least not make things much worse.

Here are the results for an update_index.lua with 48 threads for the load on the master. No commit delay was used to help the parallel replication (--binlog-commit-wait-count=0):

As can be seen, even in this workload there is significant improvement from parallel replication. Again, the improvement is particularly high when commit is expensive, due to the possibility to do the commits in parallel and amortise the cost using group commit. But even with binlog disabled and non-durable InnoDB commit, we see some improvement, though only a modest one.

Finally, to test the absolutely worst-case scenario for parallel replication, I created another workload on the master, this time with update_index.lua running with just a single thread. This way, there is absolutely no opportunity for parallel replication to execute the actual transactions in parallel, though there is still some opportunity to speed up the commit step using group commit.

Here are the results for the single-threaded update_index.lua master load:

As can be seen, even in this extreme case, we do see some speedup when commits are expensive (durable/crash-safe configuration). But when durability and crash-safety is turned off, and commits are cheap, then the overhead of the extra thread scheduling shows, and the results of parallel replication are worse, if not disastrously so. Note by the way that with this test, the time taken by the slave to replicate the load was smaller than the time for the master to generate it with single-threaded sysbench.

Conclusions

So overall, I am pleased with the results of the parallel replication code in MariaDB 10.0. Single-threaded applier has been a major limitation in replication for a long, long time, and I feel now fairly confident that this can play an important part in solving that. I hope this will inspire users to start testing their own loads and hopefully getting good results and/or reporting any issues found on the way.

To test, use MariaDB 10.0.9 or later on both master and slave. Configure the slave with --slave-parallel-threads=20 or something like that. And optionally, if your application can tolerate some extra commit latency, set some reasonable values for --binlog-commit-wait-count and --binlog-commit-wait-usec.

By the way, the raw numbers for the different runs can be seen in this mail on maria-developers@.

March 10, 2014
Peter Makholm a.k.a. brother
Peter Makholm
»Hvorfor bruger man ikke altid 8192-bit nøgler til AES og RSA?« cirka sådan lød spørgsmålet for et par dage siden. På overfladen et enkelt spørgsmål det er ligetil at besvare: Det tager for lang tid i forhold til hvad vi tror der er behov for. Men under overfladen lurer der en række lige så indl...
March 07, 2014
Ubuntu Thrusty Tahr bug hunt… (March 07, 2014, 17:59 UTC)

Ubuntu 14.04 LTS med kælenavnet Thrusty Tahr bliver efter planen udgivet om en måned og ti dage.

3518341916_87f61c997f

Den første Beta blev frigivet for lidt over en uge siden, torsdag d. 27. februar.
Hvis man har lyst til at hjælpe med at få luget de sidste bugs ud af denne Ubuntu, før udgivelsen d. 17. april, er det bare at komme i gang.

Der er flere forskellige muligheder for at hjælpe, afhængigt af ens tekniske snilde.
(Bemærk at før den endelige udgivelse må Ubuntu 14.04 stadig antages at kunne opføre sig ustabilt. Man skal således ikke bruge den til maskiner hvor man ikke vil risikere nedbrud, eller til data man ikke kan tåle at miste. Selvom Mark åbenbart tager chancen længe før udgivelsen…)

Generelt er tanken at alle bugs som bliver opdaget før udgivelsen har meget større chance for at blive rettet, så den endelige Ubuntu 14.04 kan tage sig rigtig godt ud, og ikke giver nye brugere uheldige oplevelser.

I denne weekend er der Ubuntu Bug Weekend. Hvis man ikke har en computer som man kan undvære til at teste Ubutnu Thrusty Tahr på kan man installere VirtualBox, og køre 14.04 virtuelt, for at gå på jagt efter bugs i de programmer man kender bedst, eller for at hjælpe med at få be- eller af-kræftet de bugs som allerede er blevet rapporteret.

Om ca. en måned er der Ubuntu Global Jam. I København håber vi at mødes i løbet af lørdag d. 5. april og/eller søndag d. 6. april og f.eks. arbejde med bug triaging, oversættelser, planlægning af udgivelsesfest eller hvad vi ellers har lyst til. Se denne post på vores forum, og Facebook begivenheden.

Go’ jagt!

Billede af Sankara Subramanian

March 06, 2014
Peter Toft a.k.a. pto
Git submodule HOWTO (March 06, 2014, 21:57 UTC)
Peter Toft Jeg har tidligere i dette tidligere blog-indlæg åbnet diskussionen om hvordan man med versionskontrolsystemet Git kan håndtere bånd mellem afhængige software-projekter. Jeg har sidenhen arbejdet bla. med “git submodule” og jeg er vist kommet et godt stykke rundt om det. Derfor vil jeg i dette blo...

Leave a comment

March 05, 2014
Tivoli Fredagsrock kalender 2013 (March 05, 2014, 20:58 UTC)

800px-Stine_Hjelm_Jacobsen_-_Electric_Lady_Lab_ANR_SEP2011_ubt

Edit: Jeg håber at få tilføjet koncerter for 2014 til kalenderen indenfor en uge eller så. – sbc 5. maj 2014

Tivoli laver stadig ikke en kalender til Fredagsrock som er til at importere i diverse digitale kalendere.

Så jeg har gjort det for dem.
Kalenderen for Fredagsrock 2013 kan ses nedenfor. Selvfølgelig helt uofficielt, med forbehold for trykfejl, ændringer og uden ansvar.

Direkte links til: ical .ics, html og xml

http://www.fredagsrock.dk/ er selvfølgelig stadig den officielle side for Fredagsrock (og her kan også findes info om de forskellige kunstnere).
Fredagsrock har også en Facebook side.

Foto af: Tomasz Sienicki

March 04, 2014
Poul-Henning Kamp a.k.a. phk
Poul-Henning Kamp Til at begynde med kunne enhver dandy med penge nok gøre trafikken usikker med et af disse nymodens automobiler. Mange af dem ansatte en mekaniker sammen med bilen, hvis de da ikke selv opfattede sig som sådan og stolt førte sig frem på de bonede gulve med oliesmurte fingre. Da Henry Ford havde...

Leave a comment

March 02, 2014
First Class Goto (March 02, 2014, 00:00 UTC)

Oort is an experimental programming language I have been working on, on and off (mostly off), since 2007. It is a statically typed, object-oriented, imperative language, where classes, functions and methods can be nested arbitrarily, and where functions and methods are full closures, ie., they can be stored in variables and returned from functions. The control structures are the usual ones: if, for, while, do, goto, etc.

It also has an unusual feature: goto labels are first class.

What does it mean for labels to be first class? It means two things: (1) they are lexically scoped so that they are visible from inside nested functions. This makes it possible to jump from any point in the program to any other location that is visible from that point, even if that location is in another function. And (2) labels can be used as values: They can be passed to and returned from functions and methods, and they can be stored in data structures.

As a simple example, consider a data structure with a “foreach” method that takes a callback function and calls it for every item in the data structure. In Oort this might look like this:

table: array[person_t];

table.foreach (fn (p: person_t) -> void {
        print p.name;
        print p.age;
    });

A note about syntax. In Oort, anonymous functions are defined like this:

fn (<arguments>) -> <return type> {
    ...;
}

and variables and arguments are declared like this:

<name>: <type>

so the code above defines an anonymous function that prints the name and the age of person and passes that function to the foreach method of the table.

What if we want to stop the iteration? You could have the callback return true to stop, or you could have it throw an exception. However, both methods are a little clumsy: The first because the return value might be useful for other purposes, the second because stopping the iteration isn’t really an exceptional situation.

With lexically scoped labels there is a direct solution – just use goto to jump out of the callback:

  table.foreach (fn (p: person_t) -> void {
          print p.name;
          print p.age;

          if (p.age > 50)
              goto done;
      });

@done:

Note what’s going on here: Once we find a person older than 50, we jump out of the anonymous callback and back into the enclosing function. The git tree has a running example.

Call/cc in terms of goto
In Scheme and some other languages there is a feature called call/cc, which is famous for being both powerful and mind-bending. What it does is, it takes the concept of “where we are in the program” and packages it up as a function. This function, called the continuation, is then passed to another, user-defined, function. If the user-defined function calls the continuation, the program will resume from the point where call/cc was invoked. The mind-bending part is that a continuation can be stored in data structures and called multiple times, which means the call/cc invocation can in effect return more than once.

Lexically scoped labels are at least as expressive as call/cc, because if you have them, you can write call/cc as a function:

call_cc (callback: fn (k: fn()->void)) -> void
{
    callback (fn() -> void { 
        goto current_continuation;
    });

@current_continuation:
}

Let’s see what’s going on here. A function called call_cc() is defined:

call_cc (...) -> void
{
}

This function takes another function as argument:

callback: fn (...) -> void

And that function takes the continuation as an argument:

k: fn()->void

The body of call/cc calls the callback:

callback (...);

passing an anonymous function (the continuation):

    fn() -> void {
        goto current_continuation;
    }

@current_continuation:

that just jumps to the point where call_cc returns. So when callback decides to invoke the continuation, execution will resume at the point where call_cc was invoked. Since there is nothing stopping callback from storing the continuation in a data structure or from invoking it multiple times, we have the full call/cc semantics.

Cooperative thread system
One of the examples on the Wikipedia page about call/cc is a cooperative thread system. With the call_cc function above, we could directly translate the Wikipedia code into Oort, but using the second aspect of the first-class-ness of labels – that they can be stored directly in data structures – makes it possible to write a more straightforward version:

run_list: list[label] = new list[label]();

thread_fork (child: fn() -> void)
{
    run_list.append (me);
    child();
    goto run_list.pop_head();
@me:
}

thread_yield()
{
    run_list.append (me);
    goto run_list.pop_head ();
@me:
}

thread_exit()
{
    if (!run_list.is_empty())
        goto run_list.pop_head();
    else
        process_exit();
}

The run_list variable is a list of labels containing the current positions of all the active threads. The keyword label in Oort is simply a type specifier similar to string.

To create a new thread, thread_fork first saves the position of the current thread on the list, and then it calls the child function. Similarly, thread_yield yields to another thread by saving the position of the current thread and jumping to the first label on the list. Exiting a thread consists of jumping to the first thread if there is one, and exiting the process if there isn’t.

The code above doesn’t actually run because the current Oort implementation doesn’t support genericity, but here is a somewhat uglier version that actually runs, while still demonstrating the principle.

March 01, 2014
Sune Vuorela a.k.a. pusling
Sune Vuorela

My hard-working KDE friend Mario asked me to help him to get Debian people to help him with his thesis. Here is what he writes:

Dear Free Software contributor*

I’m currently in the process of writing my diploma thesis. I’ve worked hard during the last few weeks and months on a questionnaire which shall collect some data for my thesis. Furthermore the data of this survey will be interesting for the Free Software communities as well.

So please take some time or add it to your todo list or, even better, go directly to my questionnaire and help me make a great diploma thesis and improve the Free Software community in some ways.

The questionnaire takes some 20 to 30 minutes. At the end of the questionnaire you’ll find a way to participate in a draw where you can even win something nice.

In a first round I got the feedback that the length of the questionnaire and that some questions (mostly the ones at the beginning of the questionnaire about the 12 different tasks) are quite abstract and difficult. But please try it, try your best and take the time and brain power. The remaining part of the questionnaire [1] (after these two pages with the tasks questions) is quite
easy and quickly done. And you have the possibility to come back to where you
have left filling in the questionnaire after a shorter or longer break.

And if there are any questions, feedback or you need help don’t hesitate a moment to write me an email or ping me on IRC (freenode.net and oftc.net) as unormal.

This survey will be open till Sunday, the 9th of March 2014, 23.59 UTC.

Thanks to all for reading and helping and towards the summer of 2014 you can read here what all the data you gave me showed us and where we can learn and improve.

Thanks in advance and best regards
Mario Fux

* By contributor I mean not just developers but translators, artist, usability
people, documentation writers and many more. Everybody who contributes in one
way or the other to Free Software.

Leave a comment

February 28, 2014
Unix and automation/puppet mand søges.. (February 28, 2014, 16:45 UTC)

Da vi alt for længe, har haft alt for travlt, søger EnableIT ApS en medarbejder til at afhjælpe det "problem" :)

Vi er idag 2 personer (ejerne) og du vil altså være den første medarbejder (men i og med ejerne er teknikere, så er der ikke nogen urealistiske forventninger her - håber vi :)

Arbejdsformen vil være blandet, mellem opgaver som konsulent for forskellige kunder, og udvikling af EnableIT's eget Puppet miljø - der driver forskellige kunders serverpark.

read more

Leave a comment

February 24, 2014

When your service over a period of several days is unavailable (or at least very flacky) for a number of users, it would really be appreciated if you would at least give some kind of acknowledgement of the problem, and a time frame for when you would expect the problem to be fixed.

A Twitter account that has last been updated July of last year and a wiki Status page that states “Service information available“, with latest updates from March last year is not good enough.

I enjoy my Ubuntu One file storage, but when a problem is happening for four days, without any information given to the users, I start looking around for alternatives. We all know they are out there…

(Yes, I did file a bug report…)

Poul-Henning Kamp a.k.a. phk
Hvem var Apples Eve ? (February 24, 2014, 18:11 UTC)
Poul-Henning Kamp Jeg har stirret på den der Apple SSL fejl: if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0) goto fail; goto fail; if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0) ...

Leave a comment

February 23, 2014
Bryan Østergaard a.k.a. kloeri
So I was dox'ed yesterday (February 23, 2014, 20:49 UTC)
and nobody gives a fuck.

Here's the associated spam:
14:53 < ~dd0sb0ss> rip
14:53 < ~dd0sb0ss> PARTY AT Vølundsgade 31, 3. th. 2200 København N
14:53 < ~zsasz> ur unicode is broken dd0sb0ss
14:53 < ~dd0sb0ss> fuq
14:54 < ~dd0sb0ss> THE OFFICIAL FREENODE PARTYLINE IS REACHABLE AT +4533137886
14:54 -!- dd0sb0ss was kicked from #freenode by kloeri_ [dd0sb0ss]

Ignoring the broken unicode that's actually the correct information. Well done on finding this information that has been publically available (by my own choice) for several years.

It's never been hard to find me and that's not changing in the future just because of some silly kids either. Unlike these kids I'm actually proud of what I do and I'm more than happy to stand by my actions with my real name and even address widely available.

And for all those sensible people out there just shaking your heads at this sillyness - you're welcome to visit, especially if you are interested in open source software or need a consultant on some project :) I'd suggest contacting me by email first though.

PS. Thanks to GNAA for this obvious advertising opportunity.

Leave a comment

Jesper Louis Andersen a.k.a. jlouis
Moving day (February 23, 2014, 13:17 UTC)
The reason this blog is not getting too many updates is due to me posting over on medium.com for the time. You can find me over there at this point:

https://medium.com/@jlouis666

where I mostly blog on the same things as here. The medium platform is nicer as a venue for writing text and I like the layout better, which is why I now prefer it to blogger.com.

I might still be crossposting popular posts, but I can't promise anything.

February 19, 2014
Sune Vuorela a.k.a. pusling
9.0 Zurg (February 19, 2014, 17:15 UTC)
Sune Vuorela

I welcome Debian 9.0 Zurg!

Thanks to Phoronix and to Wikipedia editor at 213.81.201.123 and 149.255.97.132 for their high quality fact checking.

Leave a comment

February 18, 2014
Poul-Henning Kamp a.k.a. phk
Corydon's nej = IT Havarikommisison en god ide ? (February 18, 2014, 22:56 UTC)
Poul-Henning Kamp Hvis jeg skal være helt ærlig, så tager jeg Finansminister Corydons modstand imod en IT havarikommission som et godt tegn. I min optik har Bjarne Corydon næsten konsekvent gjort det forkerte, hver gang han har haft lejligheden til det. Sælge DONG ? Oven i købet til de værste svindlere på Wall ...

Leave a comment

February 14, 2014
Peter Makholm a.k.a. brother
Peter Makholm
I sidste uge gjorde Andreas Steenberg, IT-ordfører for Radikale Venstre, sig bemærket i mit twitter-feed ved at påstå at NemID ikke er kritisk infrastruktur. Betyder det mon at NemID ikke er kritisk eller at det ikke er infrastruktur? Jeg er bange for at det er begrebet infrastruktur vi er uenig...
Poul-Henning Kamp a.k.a. phk
Den Gyldne Regel #2 (February 14, 2014, 09:12 UTC)
Poul-Henning Kamp Trafikbranchens håndtering af Rejsekortet her til morgen har uden tvivl været instruktiv og belærende. Tillad mig at foreslå at man lovfæster følgende princip: "Dem der har magten over et IT system, bærer alle tab det medfører." I det konkrete tilfælde ville det følge indlysende deraf, at hvis...

Leave a comment

February 13, 2014
Poul-Henning Kamp a.k.a. phk
ITHK: Sker der noget ? (February 13, 2014, 08:41 UTC)
Poul-Henning Kamp Det lader til at min ide om en IT havarikommission har slået rødder og at der er politikere der vil kigge på sagen. Helt uforbavsende er "IT-Branchen", en interesseorganisation for nogle relativt få men til gengæld meget store IT firmaer, stort set alle med udenlandsk ejerskab, ude og kalde det ...

Leave a comment

February 11, 2014
Poul-Henning Kamp a.k.a. phk
Op af stolen, stop aflytningen. (February 11, 2014, 08:31 UTC)
Poul-Henning Kamp Bare en reminder: Idag klokken 16 starter demonstrationen imod aflytningen af al teletraffik foran USAs ambassade og fortsætter derfra til Christiansborg. Dettte er ikke en partipolitisk sag, det er folket mod alle politikerne. Særligt bør dem der kalder sig "liberalister" se at komme i vandre...

Leave a comment

February 10, 2014

I have just implemented MASTER_GTID_WAIT() in MariaDB 10.0. This can be used to give a very elegant solution to the problem of stale reads in replication read-scaleout, without incuring the overheads normally associated with synchronous replication techniques. This idea came up recently in a discussion with Stephane Varoqui, and is similar to the concept of Lamport logical clock described in this Wikipedia article.

I wanted to describe this, hoping to induce people to test and maybe start using this, as it is a simple but very neat idea, actually.

A very typical use of MariaDB/MySQL replication is for read-scaleout. The application does all updates against a single master, which replicates to a set of slaves. The application can then distribute its reads among the slaves. If most of the database load is from reads, then this is an effective way to scale beyond what a single database server can handle.

The problem of stale reads occurs since MariaDB/MySQL replication is asynchronous. There is some delay from the commit of a change on the master server until that change becomes committed and visible on a slave server. This problem can be quite annoying in many cases, eg. user changes some preferences and reloads the page (read goes to a different slave server which is not caught up); now the change looks as if it was lost.

The idea is to use the MariaDB Global Transaction ID (GTID for short) as a logical clock. When we do an update on the master, we can obtain the associated GTID from @@LAST_GTID. Then on the slave, we can execute a MASTER_GTID_WAIT(<GTID from master>, <timeout>) before doing a query for which read consistency is critical; this will ensure that the slave will catch up before the query is run, and that no stale read results. Or if we get a timeout, we can try another slave server, which hopefully is not lagging as much behind.

Typically, we would pass around the GTID (the logical clock value) between different parts of the system to maintain consistency where it is needed and avoid stale reads. Eg. in a web application, we would set a cookie with the GTID when an update is made to the database. Then on the next HTTP request, we have the GTID which is needed in MASTER_GTID_WAIT(), even if it goes to a completely different web server. Or if we send an email, we can encode the GTID in any contained link, to be sure that it will work whichever slave server it might end up in, possibly in a different data center on a different continent.

By passing around the GTID whenever we communicate between parts of the system, we can ensure that if transaction A is known to have occured before transaction B, then any change made by A will also be visible to B. If there was no communication (direct or indirect), then B cannot know that A came before - and if there was communication, we can avoid stale reads by passing the GTID as part of that communication.

The great thing about this technique is that it is optional. We can use it for just the queries where avoiding stale reads is critical, and only there take the associated penalty (in the form of increased latency). Other parts of the database or application will not be negatively affected. This is much more flexible than using some form of synchronous replication, which will incur some penalty for all parts of the system.

So for applications that require more performance than what is currently possible to get from fully synchronous replication, we can choose consistency in a few critical places where it matters, and go for the performance of asynchronous replication in the rest.

I am hoping that some users will start to do tests with this, and I am very eager to hear any feedback and suggestions for further improvements (the maria-developers@ list is a good place for those). I think this can become a very useful technique in many cases.

By the way, note that it is not necessary to actually switch the replication to GTID to use this technique, as in MariaDB the GTIDs are always generated and replicated, even if they are not used for the slave when connecting to the master. This should make it simple for people to start testing. One will need to run MariaDB 10.0 on both the master and the slave, of course - but this could be achieved by setting up one MariaDB server as a slave of the main system (which could be running MySQL or Percona Server or MariaDB 5.5 or whatever), and then setting up another MariaDB 10.0 slave using the first slave as master.

If this turns out to be something that people find generally useful, it would be interesting to integrate this more tightly into the client protocol and client API. The @@LAST_GTID could be sent automatically back with the query results, the same way that @@LAST_INSERT_ID is done. And another option could enable the automatic execution of the MASTER_GTID_WAIT() call before queries sent to the slave servers. This could make the technique almost transparent for the end application, and could help avoid the overhead of extra client-server roundtrips and extra SQL parsing.

However, that is only worth doing if this turns out to be something that will actually be shown to be useful in practice. So if you want to see that happen eventually, then get testing and send in that feedback!

The code is available in the 10.0 bzr tree now, and will be in the next (10.0.9) release of MariaDB.