Billboard on music piracy: "The Crack Connection" by Tamara Conniff : really?

So I was having lunch when a friend of mine pointed out an article in Billboard titled "The Crack Connection". It was about music piracy. I took it, expecting the usual FUD straight out of RIAAs and MPAAs throat.

But this article was a jewel in it's own right. Tamara Conniff is the Executive Editor and Associate Publisher of Billboard is the one responsible for polluting the writescape with it.

In a satirical style she points out that the organized crime copying and selling music is bad bad bad and that they cost the state a lot of tax money the music industry lost in sales to these questionable individuals. So she goes (gleefully) on and puts the selling of pirated music right in the category of like selling crack, and how in 2007 selling pirated music is not any longer a misdemeanor.

I may quote here a few of the jewls.

• "Dealing pirated CDs on the street corners is more lucrative then selling crak."

• "But think again. In los Angeles, gang members and drug dealers are now setting up shop to sell CDs."

• "To deal with what has essentially become organized crime, the anti-piracy division (of los angeles) needs a bigger force. It also needs more money."

• "The anti-piracy unit has served 58 search warrants, made 190 arrests and covered more then $10.7 million in contraband for 2006. Imagine what they could do with more manpower."

• "It's killing the music business. Not just the artists---it's killing the songwirters, engineers, the recording studios, the record stores, the graphic designers who do album art, photographers, mixers, producers and the mailroom clerk."

• "If we make enough noise, maybe, just maybe more funds will get approved. To make a difference, go to riaa.com"

Dear Tamara Conniff:
It's unquestionable that criminal elements engage in all kinds of piracy, big news, tell me new, and certainly something has to be done about it.
When you told me how the music industry is beeing killed (having multi-digit billion revenues) by these petty pirates I was moved to tears over how producers, executives and studios are soon to be seen begging the streets for mercy money.
But let's look at the cold hard facts for a moment painfull as they are:

• Wouldn't you (and the content industry) missuse copyright to establish a marketing monoply to hold prices high there wouldn't be a black market.
• Would you engage your customers in innovative content services they wouldn't go out of their way to do with your content what you'd like to forbid them to. Here's a friendly cue: When your customers do something they like doing with your product, what's smarter, stamp down on it or try to develop a market on it?
  
• Do you really think that selling pirated music is about as bad for society as selling crack to kids? Well I'm sure you rather would your kids buy crack then?
• When the RIAA/MPAA wouldn't bend the laws to their hearts wishes and blow civil offences hugely out of proportion they wouldn't clog up court so badly. I mean, do you really think that your court time for sueing a Grandma and an 8year old is more important then that victims of Rape, Murder get Justice dispensed as quickly as possible?
• Nobody, and I mean NOBODY is ruining the business of the music industry then the music industry themselves. You artificially hold prices unreasonably high, you complain about your customers openly all the time, you sue a percentage of them, you go as far as to actually break the law to get what you think is yours (yes DRM is actually unconstitutional as it is today). Do you think consumers are impossible to annoy?
  
• You constantly are in the news, and almost always it's bad news (for you). In countries political parties have been formed to form a defense to what the RIAA/MPAA are mounting.... political parties, dear lady. You can tell by how annoyed youths are when they start to engage about something with politics, because that's usually the last place you see them doing anything (youths are notoriously annoyed about politics and thus usually avoid it).

So to put your picture in perspective, sure I feel for you, you have to deal with crooks and thus you go on to destroy democratic and human rights in the row and are a major moving force for trashing all what a sane, just and humane society stands for in the name of bigger profits for an already fat industry. Hell I can even sympthize, but don't expect me to shed a tear here.

Tamara, a new time is coming, and it's coming fast. You alienated your customer base from the craddle up. You missed out on the prosumer movement and soon you'll be extinct, killed by your own inevitable collapsing mass. Unable to adjust to a time of rapid change and surprising emerging markets and usages. Copyright is a phase that is going to pass as it is today, so enjoy your time in the sun while a mafia of content cartels can reap all the business from a legal situation bend to their benefits, it's not going to last forever because as other people have said before, Tamara, "we are the people".

Gamemodes

Yesterday the producer of Battlefield2 answered some questions in a chat. I particularly took notice of the following.

*Importfan85* how many different game types will there be?

[EA]ScottyRotten: Battlefield 2 is conquest only. We've been asked before why that is and the question is basically user interest. We measured the number of people playing the other game modes (Deathmatch, etc.) in BF1942 and the numbers were quite small. Therefore we decided on focusing on other parts of the game like the teamplay features.

Ok. So they looked at Battlefield 1942 and BFV, found that almost nobody played Deathmatch given a choice between Conquest and Deathmatch ( there really aren't more gamemodes ).

I can fairly understand that they scraped Deathmatch for this. But how they made the connection that Conquest would be the only thing gamers ever wanted to play is a miracle to me.

Shit infestation

Warning, if you think ATL and MFC are enjoyable API expiriences, you should not read this.

I'm working in a project making a product that needs to cross-compile against both Solars and Windows. For this you obviously need unified interfaces for containers.

Trough some history I'm glad not to have been involved, the choice fell on the Microsoft flavor of containers and API's, coming from ATL and MFC.

Obviously I'm not a fan of these APIs. I'm hardly a fan of the STL, but I hink its interface is as close as you get to coherent and nice as is possible in a language like C++.
Of course I think coherent and nice are important properties in code, if you disagree there's again no point in reading along, go troll some Blogs about how great Windows XP is.

The choice to take proprietary containers from M$ not only led to the lamentable situation that these actually had to be implemented on unix by this project. It also lead to a complete mess whenever you start using std:: containers. For this reason std:: containers are shunned, and everything that works on them.
If you don't know ATL, rest assured, iterators are not supported.
No! You can't go and iterate like for( contType::iterator i=x.beg(); i!=x.end(); i++ ) (of course this is a far cry from "for x in y" of python) but hey, as good as it gets aye?

Well the point of this post? I'm annoyed. Even though I'm not on windows, and even though what we programm right now is never going to work on windows, I still have to cope with their moronic excuse of containers.

That's what the natives from where I come call a shit infestation.

Battlefield 2 Modding

I'm toying with modding Battlefield 2, and for that purpose I'm putting up a reference page of it.

It's a lot of work specially documenting the stuff that isn't there as source. I also noticed that there's rather an abundance of API's the engine exposes, that aren't used at ally by any python.
I also came by some classes that seem to be created dynamically by C++ code, or be plugged together by other means. Thinking about it now, I've the suspicion more then what is visibile already in the Demo is scripted in python. I'll try to explore that later on.

Battlefield 2

This post is going to be about three things: Battlefield 2, my new hardware to play it and the python embedding they do.

Battlefield 2
This is simply an awesome game. A demo has just come out, and it is in many many aspects so much more and better then the previous battlefield. There are some issues I have, which would namely be the ingame browser and a bit of the loading slowness spread everywhere. But apart from that it's absolutely superb.

Battlefield 2 hardware
I was merrily happy before with a geforce 5900ultra on a nforce2 board. So this time I went for an nforce4 board. I was buying the geforce 6800ultra. At the time I decided cutting back for the processor is pointless, and bought a Clawhammer fx-55. Beeing trough that decision I thought it's silly to cut back at all, and bought the second 6800ultra for the SLI setup.
Bear with me, I know that for some things the SLI setup is a bit slower then the fastest ATI cards, but there's many considerations for me why I buy only geforces. I won't bore you however with them. I needn't be told either that the price/performance comparision for this setup is quite shitty.
Well, a friend of mine summarized it pretty good. "Dude you've just spend 400$ on a graphics card, and you've a surplus of fourty times that money, run for it or let it be!"
I'm still amazed about my own buying decisions not based on any practical considerations :D

Battlefield 2 and python
I was amazed to see that to some extend battlefield 2 uses python. I'm still figuring out the details, and digging my way trough the classes and interfaces. I have quite high hopes that this will make for some very interesting modding.

Full text indexer

It took me 15 minutes today to write a simple full-text indexer for source code, using shlex and zodb. It accesses a 20Mb database below a notable delay. It will save me endless hours of using find/grep on the ~3600 header files I find myself searching pretty often.

Implementation time: 15 minutes. Retriever 15 lines of code. Indexer 37 lines of code. Time not spend using find/grep, priceless.

Negative reinforcement works!

Yet again I work in a C++ project. The deadline looms closer. We've no working system, a lack of specification, people working on their modules in autism mode and general confusion about what we're doing anyway.
Personally I've been struggling with an impossibly retarded build-system for the past three weeks, I tried to get the "big picture" to compile so I can work against something that runs. I too will resort to implement an interfaces that understand the protocol instead of trying to integrate.
Ahead of us lays undoudbtedly a big-crunch integration-havoc and possibly worse things.

Blame can be laid partly to a development/source control system that easely breaks, and is very slow to operate due to sheer size and bad setup. ( honorable mention of sniff+, Sun Workshop and Synergy CMS )

Part of the blame firmly rests at the fact that 5 weeks before the deadline nobody who worked on the previous prototype is present, the chief developer has just taken off on a 3 weeks vacation and 20% of the team are noobs to this project.

We ( meaning the noobs, including me ) still struggle to grasp the system, while our deadline is 2 1/2 weeks ahead of us.

Regardless of expirience in this project, we all have no clue what's supposed to happen and how we get there.

Oh, and I should mention that I've two deadlines, one for my implementation task, and one for the completion of something completely different ( packaging stuff we should assume responsibility for after customer decided that this should be our task ).
Lo and behold, both deadlines accidentally fall to the same day. Of course for the packaging stuff I'm equally clueless as anybody, and the guy who did this before will embark on a three-week vacation a week before the deadline.

Sometimes I figure I got the short-straw in the game, whatever it is. Ah that reminds me of the fact that I still have 5 weeks of vacation to kill this year. I wonder if this would be a good time...

Darwin Award

My todays nominees would be the people who try to fly in the wheel well of an airplane.
Obviously there's multiple dangers awaiting the eager well-traveler there.

-You can get crushed by the wheels
-You can get crushed by the doors closing/opening
-You can freeze do death
-Die of oxygen deprivation in airy hights
-You can fall do death
-Slide to death ( in case you fell out at a couple of hundreds of miles per second while the airplane is still on ground )

Surprisingly as it sounds, there seem to be records of hundreds such incidents, few of which survived.

I don't know what's mor baffling, how people actually find themselves doing such an incredibly stupid thing, or that some of em actually manage to survive.

ElementTree

I've said this before, and I still think it holds true. ElementTree is very nice.

There was a recent discussion about it on comp.text.xml, which got the attention of Fredrik Lundth

I do much apreciate ElementTree. Of course using a non-DOM API to process xml conviniently is heresy, and will call the gods crushing down on us.

I find the proponents to a strict and true die hard one Document Model API hillarious in their self imposed blindness towards pragmatism.

But then, it's not my time I'm wasting on cumbersome and broken API's, or is it? Of course the Net-dammage in the form of popular implementations of DOM can't be taken into account lightly...

PyPy 0.6 out

Alright, now this one gets me quite excited. Just 5 minutes ago I argued with a friend of mine, that PyPy is in fact going forward, and he argued that they'll not have a release anytime < 10 years.

However, there is a release 0.6 now. From the tutorials and explanations, there's plenty a mention that this is higly experimental. I can spot type inference there, as well as translation to machine code ( or at least something very very close ).

This is exiting because once PyPy, executed on the CPython can translate to machine code, it could compile itself to a running programm. One which in turn can again compile future versions of itself and so forth.

Not only does it mean that python gets very flexible with this, it also means that integration with native code, runtime speed and a ton of other things get very fancy now.

Onwards to world Domination with PyPy ;),
Kudos and good Karma for the group around PyPy, you do an awesome job!

RPC and protocol bloat.

While implementing rpc stuff, I noticed that the protocol bloats. Here's an example.

remote.text( 's' ) becomes ('text', 0, ('s',)) which becomes some 40 or so bytes once marshalled. A byte id and a single byte for the letter would be 2 bytes in contrast.
The problem about ('text', 0, ('s',)) is that it has more information. It carries the method to be invoked remotely, what type of command it is ( i.e. a call and not an error message ), and the argument-list for the remote method in an easely digestible form.

I realize I could optimize somewhat with a fancy marshaller, that negotiates formats with foreign marshallers, like templates or identifier lookup lists.

But then let's say each tiny message translates to abou 50 byte. a user can only type around 2 baud, so the generated netflow would be around 100baud/user. you can support at least 128 users on a 24kB modem line then ( assuming they do not do worse things then typing and pasting short text ).

Is optimizing at this point really worth it?

I guess I should just encapsulate properly, and later if I really want I can still try to optimize ('text', 0, ('s',)) structures to a few bytes.

RPC, Nodes'n stuff

I've been hacking away at an rpc mechanism. Fully knowing that I ignored proper asynchronous dispatching, just a very thin layer over messaging. However, during the process I noticed that it would be rather intersting to leave the client/server pattern, and instead provide a generic object that is a node which can both accept and build connections. I'm not really sure how a node will find out in a nice way where to connect to, but I think it's interesting none the less.

Networking

I've been on it again. I don't know why precisely I torture myself with network programming. There's a pleathora of toolkits for precisely that readily avaiable.
Generally I do scratch-writes of existing libraries only 1/4 until I grasp the concept fully and convert to an existing library.

The things I've had direct exposure to where asyncore, medusa, twisted, xmlrpc, tcpserver and of course socket. I think there's far more, but these are the ones that notably stick around my internal memory so they might deserve the mention, honorary at least.

I can' really reason that any of those above would be insufficient or somehow lacking, they just do not really apeal to me.

So, I'm on the fourth ( or was it the fifth ) rewrite of my network library. Somewhere at the second rewrite I decided asynchronous can't be done properly, and sacrified it for threads. Meanwhile having aquired more network-karma, I actually reversed my opinion, and think asynchronous can in fact be done properly, and it will be a big merrit in the end.

Of course I reinvent all sorts of wheels in the process, including but not limited to rpc mechanisms, message transport protocols and asynchronous messaging.

I still don't know really why I do it. But there's only two possible explanations. Either all existing stuff sucks beyond comic relief, or I've still not grasped the fundamentals of network programming, even after years of exposure.

I fear the latter is far more likely, but duh... can that really be?

Scheduling

It was monday afternoon. A friend of mine aproached me with a seemingly simple problem. The problem is as follows:

-There's a number of tasks, that have to be repetetly called each second. The amount of each task's repetition is fixed, and has to be met.
-Some tasks have a dependency, and thus execution order for those matters.
-One task may be used as a fill-up, when there's time left, or to drop if there is to little time.
-All tasks should be evenly distributed, in order to make the system apear as fluent as possible.

( note, these requirements are already a bogged-down version, that tries to eliminate a ton of complexities, or so we thought )

As the day progressed, and we talked for hours and wrote code and tested, it turned out this isn't a simple problem, but a hard one. In fact, we ended up with something that performed bad, and on top of that couldn't fill-up properly.

I told my friend up-front that I think that this is not one of the simple problems, vaguely remembering some school lessons about operating systems, and process scheduling. However, we tried to adjust requirements to a form that made it all implementable, seemingly, just that we ended up having still difficulties getting it right.

It was kinda Bummer, I feel I need to ponder this problem longer, and understand more of the theory to get it precisely right. I hope I can come up with something vastly better when I get to the third or fourth rewrite.

Patching

Once in a while a pc-game hits the market that needs patching badly from the get go. In the best of cases the patches come fast, eventually they come before the game is released. The producers of such games tend to fall short in favor of critical customers, while the press joins the chorus. I'm not sure if this is the right kind of bad publicity that's as good as good one.

On any account, I'm baffled by the producers of said pc-games, because they could easely avoid the trouble. Instead of having customers find, download and install a patch ( by which time even the very simple minded start to get clues ), they could just incorporate automatic-patching.

Without going in to technobabble, just give the user a cute little box, offering him to update/restart the game when an update is avaiable ( if you feel brave you could also try live-patching, warning though, not for the faint of heart ).

Chances are your user will just klick yes without reading, that's about 70% people less that are even noticing you're patching the game short after you released it. And the remaining 30% will be much less enraged because the actual efford they had to spend to get the patch migitates their feel to have been screwed by you. I even think that automatic-patching is percieved as quite normal, whereas hand downloadable patches are precieved as literally patchwork philosophy ( altough technically it's both the same ).

Make Wrong Code Not Compile/Inteprete

I like Joels latest article about making Wrong code look wrong. The whole shebang about making relationships as visible as possible and stuff.

I still do not like Hungarian notation and I do not agree with his assertion of the best solution.

The first reason is that it is dependent on a reader to tell an error. Everbody likes errors, and I like them best when they do not interprete/compile.

Which leads me to my second reason. The hungarian notations solves problems that can be adressed by other means as well. For instance type-subclassing. However this hasn’t deserved a mention in Joels post, which I find somewhat dissapointing.

Normally code is litered with dependencies, intricate semantics and stuff that makes it harder then to write blog entries. I’m pretty glad the less there is to worry about and the more is automatically checked. Hungarian notation introduces another layer of semantics on top of the existing complexities, I find it distracting.

The next reason I dislike Hungarian notation is that it forces you either to use acronyms in names, or waste rediculus amounts of name-estate.

I dislike acronyms in names because it is an additional inderection, with which I’m forced to deal in order not to make errors. Indirection because extensive use of acronyms ( as in hungarian notation ) leads to the code beeing litered with acronyms, and each of it means something that’s supposed to tell you something, that you have to remember in order to find out what it means, instead of telling this to you up front.

Obviously screen estate is limited, and so is name-estate. I like to spend name-estate on reasonably describtive multi-word names ( dislike commenting the obvious really, if I can avoid it with good naming ).
Hungarian notation can’t do that for me. But it can harm my ability to form good names, by introducing more bytes stacked on top of my names, and thus taking away valuable name-estate from me.

Firewall admins, Corporate Logic

Dear reader ( if any ) I am forced to post this comment in IExploder, and do the rest of my surfing using this godawfull insecure browser. Why you might ask? The oh-so-wise(tm) firewall admins at our company obviously decided that two critical security holes in Firefox are enough to filter out any traffic going on using the UserAgent: Mozilla*.

Ahyes, this makes sense(tm) to me too, whatever happend to the dozens of unpatched critical security holes in IExploder, and it is preinstalled on our machines, traffic using UserAgent: IExploder/Malware Magnet is still alowed...

I think it's time to sigh in dispair(tm)

MetaGer - Does software move forwards?

This is an interesting article

Recap:
Pupil found paper of head of a research facility, about the next generation search engine. At the time of the writing it was not feasible mainly due to network bandwith constraints. Pupil cleverly implemented the idea salvaging his existing broadband account, and now everybody's abuzz.

What struck me about this is that said pupil not only implemented the whole thing from scratch, he also found a way to cleverly implement the new features in a way that is feasible today. And all he needed was a describtion of the idea.

Amazing feat! It will not surprise you much that the pupil used python for this. In face of this, I suspect that under the right conditions python's not only more productive, but by orders of magnitude so.

Texture Packing

I toyed a round a bit with texture-packing. The idea is to pack multiple arbitary sized textures on a bigger one. In the context of a sprite engine, this makes sense. Today when you write games, whatever you do, you use the exisitng 3d hardware, even for 2d stuff. The thing you want to avoid when rendering with super-fast hardware is shuffling data around between card and ram. Thus it's a sensible aproach to load all textures needed for a batch of work once, and then have primitives use them.

Anyway, here's a pretty pic :D

( p.s. of course this is all written in python, source follow later )

dictionary speed

This entry got me thinking.

Here's my 5 cents. In all cases the construct

if key in dictionary:
    value = dictionary[key]
else:
    value = default

performed best.

update: I increased the amount of data to reduce the amount of white-noise in the test results, though things haven't changed. An explanation to the output. The programm compares nine data-sets against four test functions. The nine data sets are the combinations of size-distribution between search/dictionary and size-distribution between search and matching keysets ( thereof the small/even/big combinations ).

update2: Why, the version of python is 2.4 of course :)

Here's the output of the programm.

big dict, small search, big intersect
 test_has_key         0.9690
 test_get             1.0000
 test_try             1.2190
 test_in              0.5930
big dict, small search, small intersect
 test_has_key         0.8750
 test_get             1.0320
 test_try             6.6250
 test_in              0.5150
big dict, small search, half intersect
 test_has_key         0.9220
 test_get             1.0310
 test_try             3.9530
 test_in              0.5470
even, big intersect
 test_has_key         1.4370
 test_get             1.4840
 test_try             1.7030
 test_in              1.0780
even, small intersect
 test_has_key         1.4530
 test_get             1.6090
 test_try             7.3120
 test_in              1.0780
even, half intersect
 test_has_key         1.4690
 test_get             1.5620
 test_try             4.5630
 test_in              1.0780
small dict, big search, big intersect
 test_has_key         0.1090
 test_get             0.1250
 test_try             0.7970
 test_in              0.0630
small dict, big search, small intersect
 test_has_key         1.0790
 test_get             1.2180
 test_try             7.0160
 test_in              0.7340
small dict, big search, half intersect
 test_has_key         0.5790
 test_get             0.6560
 test_try             3.8440
 test_in              0.3910

and that's the programm

import sys, time

def random_key( keylen = 10 ):
    import random
    return ''.join(chr(random.randint(ord('a'),ord('z'))) for _ in range( keylen ))

def new_key( existing_keys ):
    key = random_key()
    while key in existing_keys:
        key = random_key()
    return key

def unique_keys( amount=1000, existing_keys=set() ):
    keys = set( existing_keys )
    for _ in xrange( amount ):
        key = new_key( keys )
        keys.add( key )
    return keys-existing_keys

def data( dictionary_size=1000, search_size=3000, intersection=0.5 ):
    dict_keys = unique_keys( dictionary_size )
    intersection_amount = int(search_size*intersection)
    unique_search_keys = unique_keys( search_size-intersection_amount, dict_keys )
    intersection_search_keys = set(list(dict_keys)[0:intersection_amount])
    return dict((key,None) for key in dict_keys), unique_search_keys | intersection_search_keys


def core_test( out_prefix, dictionary, searched, test_functions ):
    print out_prefix
    for function in test_functions:
        start = time.time()
        for _ in range( 20*(len(dictionary)/len(searched) or 1) ):
            function( dictionary, searched, None )
        end = time.time()
        print '\t%-20s %2.4f'%(function.__name__, end-start)

def inner_test( out_prefix, dictionary_size, search_size, test_functions ):
    #big intersection
    dictionary, search = data( dictionary_size, search_size, 0.9 )
    core_test( '%s, big intersect'%out_prefix, dictionary, search, test_functions )
    #small intersection
    dictionary, search = data( dictionary_size, search_size, 0.1 )
    core_test( '%s, small intersect'%out_prefix, dictionary, search, test_functions )
    #half intersection
    dictionary, search = data( dictionary_size, search_size, 0.5 )
    core_test( '%s, half intersect'%out_prefix, dictionary, search, test_functions )


#test functions
def test_has_key( dictionary, searched, default ):
    for key in searched:
        if dictionary.has_key( key ):
            value = dictionary[key]
        else:
            value = default

def test_get( dictionary, searched, default ):
    for key in searched:
        value = dictionary.get(key, default)

def test_try( dictionary, searched, default ):
    for key in searched:
        try:
            value = dictionary[key]
        except KeyError:
            value = default

def test_in( dictionary, searched, default ):
    for key in searched:
        if key in dictionary:
            value = dictionary[key]
        else:
            value = default

def test():
    factor = 100
    test_functions = [test_has_key, test_get, test_try, test_in]
    #big dict, small search, small
    inner_test( 'big dict, small search', 1000*factor, 1*factor, test_functions )

    #even dict/search
    inner_test( 'even', 1000*factor, 1000*factor, test_functions )

    #small dict, big search
    inner_test( 'small dict, big search', 1*factor, 1000*factor, test_functions )

if __name__ == '__main__':
    test()