The plate is bad

Christmas presents

2011-12-23T13:26:00.000+01:00

Jérôme Pansanel has completed the new serialization code for mychem and pgchem, so there is no need to handle stereo and non-stereo queries differently anymore. I have moved the index functions to GCC's vector extensions where applicable, and the first result is that index build times have been roughly cut by half while substructure search times have improved, but not that much.

Index build times

System	Index build time
pgchem with OpenBabel or Indigo	352137 ms
pgchem with OpenBabel or Indigo vectorized	192815 ms

OpenBabel with binary storage and FP2 fingerprint vectorized

Query	Hits	no Index	Hits	with Index
GH2	4840	98416 ms	4840	17044 ms
GH7	260	94053 ms	260	1564 ms
GH13	580	113690 ms	580	34504 ms
GH16	26910	99365 ms	26910	55154 ms

Merry Christmas and a happy new year!

Benchmark data published

2011-12-17T16:10:00.001+01:00

The base data used in the previous benchmarks can be found here: pubchem_0_100000.

Selected GH17 results for 10^6 structures

2011-12-15T11:04:00.000+01:00

GH17 substructure search speed

OpenBabel with binary+SMILES storage and FP2 fingerprint

Query	Hits	no Index	Hits	with Index
GH2	4840	108807 ms	4840	21164 ms
GH7	260	105050 ms	260	1934 ms
GH13	580	118978 ms	580	52416 ms
GH16	26910	109886 ms	26910	64742 ms

Indigo with binary storage and ext+sub fingerprint

Query	Hits	no Index	Hits	with Index
GH2	4840	213075 ms	4840	27887 ms
GH7	410	178963 ms	410	4451 ms
GH13	580	251938 ms	580	39134 ms
GH16	27100	172534 ms	27100	80523 ms

Bingo 1.7beta2 with molfiles as text storage

Query	Hits	no Index	Hits	with Index
GH2	4710	647889 ms	4710	21733 ms
GH7	410	538784 ms	410	6658 ms
GH13	580	675093 ms	580	12418 ms
GH16	27100	528891 ms	27100	28541 ms

Index build times

System	Index build time
pgchem with OpenBabel or Indigo	352137 ms
Bingo	3458681ms

Again, Bingo without it's index is apparently killed by the overhead of parsing text into the internal molecule format. With index it's a mixed bag, while it shines at GH13 and GH16, pgchem is about equal or faster at GH2 and GH7.

GH17 results

2011-12-14T08:56:00.000+01:00

Mikhail Rybalkin from GGA Software asked me to do this, so here it is...

The GH17 test queries used

There is a small set of queries used in article Chemical substructure search in SQL by Golovin and Henrick.These queries were lately reused in other articles:

GH1 ONC1CC(C(O)C1O)[n]2cnc3c(NC4CC4)ncnc23
GH2 Nc1ncnc2[n]cnc12
GH3 CNc1ncnc2[n](C)cnc12
GH4 Nc1ncnc2[n](cnc12)C3CCCC3
GH5 CC12CCC3C(CCC4=CC(O)CCC34C)C1CCC2
GH6 OC2=CC(=O)c1c(cccc1)O2
GH7 Nc1nnc(S)s1
GH8 C1C2SCCN2C1
GH9 CP(O)(O)=O
GH10 CCCCCP(O)(O)=O
GH11 N2CCC13CCCCC1C2Cc4c3cccc4
GH12 s1cncc1
GH13 C34CCC1C(CCC2CC(=O)CCC12)C3CCC4
GH14 CCCCCCCCCCCP(O)(O)=O
GH15 CC1CCCC1
GH16 CCC1CCCC1
GH17 CCCC1CCCC1

GH17 substructure search speed

OpenBabel with binary+SMILES storage and FP2 fingerprint

Query	Hits	no Index	Hits	with Index
GH1	0	9517 ms	0	25 ms
GH2	484	8519 ms	484	111 ms
GH3	63	8632 ms	63	43 ms
GH4	5	8950 ms	5	48 ms
GH5	36	10020 ms	36	78 ms
GH6	0	8696 ms	0	32 ms
GH7	26	8279 ms	26	31 ms
GH8	170	8454 ms	170	56 ms
GH9	348	8068 ms	348	71 ms
GH10	36	8820 ms	36	21 ms
GH11	66	9113 ms	66	52 ms
GH12	831	7920 ms	831	124 ms
GH13	58	9864 ms	58	448 ms
GH14	4	9998 ms	4	36 ms
GH15	3008	8549 ms	3008	555 ms
GH16	2691	8665 ms	2691	501 ms
GH17	2290	8717 ms	2290	560 ms

Indigo with binary storage and ext+sub fingerprint

Query	Hits	no Index	Hits	with Index
GH1	0	28161 ms	0	32 ms
GH2	484	19188 ms	484	187 ms
GH3	68	20498 ms	68	78 ms
GH4	5	23388 ms	5	33 ms
GH5	36	25887 ms	36	62 ms
GH6	0	21034 ms	0	31 ms
GH7	41	16302 ms	41	31 ms
GH8	170	16629 ms	170	78 ms
GH9	373	14005 ms	373	98 ms
GH10	37	16881 ms	37	21 ms
GH11	66	24091 ms	66	78 ms
GH12	829	14842 ms	829	210 ms
GH13	58	24470 ms	58	133 ms
GH14	4	21403 ms	4	36 ms
GH15	3047	15817 ms	3047	749 ms
GH16	2710	16767 ms	2710	732 ms
GH17	2304	17524 ms	2304	788 ms

Bingo 1.7beta2 with molfiles as text storage

Query	Hits	no Index	Hits	with Index
GH1	0	70277 ms	0	125 ms
GH2	471	56821 ms	471	156 ms
GH3	68	57754 ms	68	125 ms
GH4	5	60067 ms	5	125 ms
GH5	36	65586 ms	36	140 ms
GH6	79	57188 ms	79	125 ms
GH7	41	52134 ms	41	125 ms
GH8	170	51685 ms	170	140 ms
GH9	373	47613 ms	373	138 ms
GH10	37	49961 ms	37	110 ms
GH11	66	61176 ms	66	125 ms
GH12	774	50281 ms	829	156 ms
GH13	58	61108 ms	58	156 ms
GH14	4	53636 ms	4	140 ms
GH15	3047	50213 ms	3047	343 ms
GH16	2710	51227 ms	2710	327 ms
GH17	2304	51495 ms	2304	362 ms

Fingerprint efficiency (with regard to false positives)

FP2

Query	Candidates screened	Hits matched	false positives	Efficiency
GH1	0	0	0	1.000
GH2	485	484	1	0.998
GH3	69	63	6	0.913
GH4	37	5	32	0.135
GH5	120	36	84	0.300
GH6	79	0	79	0.000
GH7	41	26	15	0.634
GH8	177	170	7	0.960
GH9	377	348	29	0.923
GH10	37	36	1	0.973
GH11	123	66	1	0.537
GH12	831	831	0	1.000
GH13	1346	58	1288	0.043
GH14	20	4	16	0.200
GH15	3760	3008	752	0.800
GH16	3305	2691	614	0.814
GH17	3305	2290	715	0.762

ext+sub

Query	Candidates screened	Hits matched	false positives	Efficiency
GH1	0	0	0	1.000
GH2	484	484	1	1.000
GH3	68	68	0	1.000
GH4	5	5	0	1.000
GH5	47	36	11	0.766
GH6	0	0	0	1.000
GH7	41	41	0	1.000
GH8	170	170	0	1.000
GH9	373	373	0	1.000
GH10	37	37	0	1.000
GH11	66	66	0	1.000
GH12	829	829	0	1.000
GH13	259	58	201	0.224
GH14	20	4	16	0.200
GH15	3061	3047	14	0.995
GH16	2720	2710	10	0.996
GH17	2720	2304	416	0.847

Index build times

System	Index build time
pgchem with OpenBabel or Indigo	25690 ms
Bingo	336319 ms

Indigo's ext+sub fingerprint is truly more selective than FP2. Still, OpenBabel with binary storage shows the better prformance because of its ~~faster matcher~~ lower query overhead.

Also, the result for GH3, GH7, GH9, GH12, GH15, GH16, and GH17 are different between OpenBabel and Indigo, and Bingo finds 79 hits for GH6 where pgchem finds zero.

Index building on pgchem is 13 times faster than Bingo, but since pgchem (currently) does not support features like tautomer searching or SMARTS searching with index support this comparison is a bit like apples and oranges.

The slow performance of Bingo without index, comparable to pgchem without binary storage, is quite likely a result of the storage of molecules in textual representation. Parsing text to binary molecules is a first class performance killer. Unfortunately, there is no way to convert molecules into native format directly with Bingo for PostgreSQL, but Bingo does the conversion implicitly when building the index.

OBMol de-/serialization revisited

2011-12-09T15:03:00.001+01:00

The current serialization/deserialization mechanism in pgchem and mychem does not preserve stereochemistry of OBMol objects properly. As Tim Vandermeersch wrote:

The OBChiralData isn't used anymore. Also the functionsOBAtom::IsClockwise, and OBAtom::IsAntiClockwise are obsolate.Instead, you should serialize the OBCisTransStereo andOBTetrahedralStereo data objects.

Fortunately (for me, since I don't have the time at the moment), Jérôme Pansanel has:

...started the serialization of the OBCisTransStereoand OBTetrahedralStereo objects.

For the time being, I have tried the following workaround and it seems to be working well. First, I have removed now unneccessary code from the unserialization:


bool unserializeOBMol(OBBase* pOb, const char *serializedInput)
{
  OBMol* pmol = pOb->CastAndClear<OBMol>();
  OBMol &mol = *pmol;
  unsigned int i,natoms,nbonds;

  unsigned int *intptr = (unsigned int*) serializedInput;

  ++intptr;

  natoms = *intptr;

  ++intptr;

  nbonds = *intptr;

  ++intptr;

  _ATOM *atomptr = (_ATOM*) intptr;

  mol.ReserveAtoms(natoms);

  OBAtom atom;
  int stereo;

  for (i = 1; i <= natoms; i++) {
    atom.SetIdx(atomptr->idx);
    atom.SetHyb(atomptr->hybridization);
    atom.SetAtomicNum((int) atomptr->atomicnum);
    atom.SetIsotope((unsigned int) atomptr->isotope);
    atom.SetFormalCharge((int) atomptr->formalcharge);
    stereo = atomptr->stereo;

    if(stereo == 3) {
      atom.SetChiral();
    }

    atom.SetSpinMultiplicity((short) atomptr->spinmultiplicity);

    if(atomptr->aromatic != 0) {
      atom.SetAromatic();
    }

    if (!mol.AddAtom(atom)) {
      return false;
    }

    atom.Clear();

    ++atomptr;
  }

  _BOND *bondptr = (_BOND*) atomptr;

  unsigned int start,end,order,flags;

  for (i = 0;i < nbonds;i++) {
    flags = 0;

    start = bondptr->beginidx;
    end = bondptr->endidx;
    order = (int) bondptr->order;

    if (start == 0 || end == 0 || order == 0 || start > natoms || end > natoms) {
      return false;
    }

    order = (unsigned int) (order == 4) ? 5 : order;

    stereo = bondptr->stereo;

    if (stereo) {
      if (stereo == 1) {
        flags |= OB_WEDGE_BOND;
      }
      if (stereo == 6) {
        flags |= OB_HASH_BOND;
      }
    }

    if (bondptr->aromatic != 0) {
      flags |= OB_AROMATIC_BOND;
    }

    if (!mol.AddBond(start,end,order,flags)) {
      return false;
    }

    ++bondptr;
  }

  intptr = (unsigned int*) bondptr;

  mol.SetAromaticPerceived();
  mol.SetKekulePerceived();

  return true;
}

Then, when matching, I check if the query might contain tetrahedral stereo. If yes, I build the target OBMol fresh from SMILES, if no directly from the serialized object:


 if (strchr (querysmi, '@') != NULL)
    {
        //Match against an OBMol generated from SMILES
    }
    else
    {
         //Match against an OBMol deserialized from binary
    }

First Light

2011-12-06T12:18:00.001+01:00

As promised, here the first results for substructure searching c1ccccc1Cl on the first 100000 compounds from Pubchem: select * from pubchem.compound where compound >= 'c1ccccc1Cl'::molecule

Substructure search speed

Rank	Build	Storage	Fingerprint	Hits	no Index	Hits	with Index
1	OpenBabel	binary+SMILES	FP2	8070	9236 ms	8067	936 ms
2	Indigo	binary	ext+sub	8049	24821 ms	8049	2418 ms
3	OpenBabel	SMILES	FP2	8070	57971 ms	8067	5432 ms

I've checked why the OpenBabel FP2 fingerprint eliminates three structures that otherwise would pass: Using the VF2 OBIsomorphismMapper instead of the OBSmartsPattern, it's also 8067 without index. But it's about four times slower, 90526 ms without index, 8798 with index.
The structures in question are: 80944, 83450 and 99925 and I'm pretty sure it's caused by differences in aromaticity detection.

Stereochemistry in substructure searches

Check	Query	Expected	Indigo binary	OpenBabel SMILES	OpenBabel binary
R/S different	select 'C[C@H]([C@@H](C(=O)O)N)O'::molecule <= 'O=C(O)[C@H](N)[C@@H](O)C'::molecule	false	pass	fail	fail
R/S same	select 'C[C@H]([C@@H](C(=O)O)N)O'::molecule <= 'C[C@H]([C@@H](C(=O)O)N)O'::molecule	true	pass	pass	pass
E/Z different	select 'C(=C\Cl)\Cl'::molecule <= 'C(=C/Cl)\Cl'::molecule	false	pass	fail	fail

OpenBabel fails the 'E/Z different' and 'R/S different' checks, but these are known issues up to version 2.3.1. ~~More disturbing are the issues with 'R/S different'. For SMILES I can reproduce it with obgrep, so it's not my code causing it.~~

Indigo has no obvious issues with matching R/S and E/Z stereochemistry. ~~Speedwise, OpenBabel with binary storage would be the leader of the pack, but has inconsistent behaviour compared with it's SMILES storage sibling.~~ So the serialization/deserialization code clearly is missing something. But I've found a simple workaround: If the query contains chirality information it uses SMILES, otherwise binary. Speedwise, OpenBabel with binary storage now is the leader of the pack.

Fingerprint efficiency

Fingerprint type	Candidates screened	Hits matched	false positives	Efficiency
ext+sub	8090	8049	41	0.995
FP2	8145	8067	78	0.99

Pretty close.

What's up, doc?

2011-11-30T15:39:00.001+01:00

OK, I have now a working (i.e. not crashing & giving meaningful results) hybrid version of pgchem::tigress with OpenBabel and Indigo, where the OpenBabel functions in the search engine have been replaced by Indigo functionality.

Why?

To see if it can be done :-)
To learn more about programming with Indigo
To see which is faster
Indigo has better reaction support as it seems

Interestingly I could keep the external interface almost stable. As only a dozen or so functions outside the search engine are affected by the change, I think with some #IFDEFs I can make a common codebase that compiles either to 100% OpenBabel or the hybrid version.

R-Matrix implementation with Indigo

2011-11-14T10:09:00.001+01:00

Long time no see.

I have been trying programming with Indigo recently and here is my first, straightforward (to avoid the term 'naive') implementation of an Ugi like R-Matrix (Gasteiger/Engel, Chemoinformatics, 2003, Wiley-VCH, p. 185ff.) for adding make/break/change bond detection to its reaction mapper:


#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include "indigo.h"

static inline int mp(const int column, const int row, const int maxRows)
{
    return column-1+((row-1)*maxRows);
}

int main(int argc, char *argv[])
{

    int index, reaction, reactant, product, atom, bond, atomIter, reactantIter, productIter, neighborIter, totalAtoms, mappingNumber, targetMappingNumber, bondMakeBreak;
    int *bMatrix, *eMatrix;//, *rMatrix;

    totalAtoms=0;

    qword session = indigoAllocSessionId();

    indigoSetSessionId(session);

    printf("%s\n",indigoVersion());

    reaction = indigoLoadReactionFromFile("c:/temp/testrxn.rxn");

    //indigoFoldHydrogens(handle);

    indigoDearomatize(reaction);

    //printf("1:%s\n",indigoGetLastError());

    indigoAutomap(reaction,"discard");

    reactantIter = indigoIterateReactants(reaction);

    while((reactant = indigoNext(reactantIter)))
    {
        totalAtoms+=indigoCountAtoms(reactant);
    }

    indigoFree(reactantIter);

    bMatrix = (int*) calloc(totalAtoms*totalAtoms,sizeof(int));
    eMatrix = (int*) calloc(totalAtoms*totalAtoms,sizeof(int));
    //rMatrix = (int*) calloc(totalAtoms*totalAtoms,sizeof(int));

    reactantIter = indigoIterateReactants(reaction);

    while((reactant = indigoNext(reactantIter)))
    {
        atomIter = indigoIterateAtoms(reactant);
        while((atom = indigoNext(atomIter)))
        {
            mappingNumber = indigoGetAtomMappingNumber(reaction,atom);
            //printf("%i\n",mappingNumber);
            //indigoGetExplicitValence(atom, &valence);
            //printf("%i\n",valence);
            //bMatrix[mp(mappingNumber, mappingNumber, totalAtoms)]=valence;

            neighborIter = indigoIterateNeighbors(atom);

            while((atom = indigoNext(neighborIter)))
            {
                targetMappingNumber = indigoGetAtomMappingNumber(reaction,atom);

                index = mp(mappingNumber, targetMappingNumber, totalAtoms);

                if(bMatrix[index]==0)
                {
                    bond=indigoBond(atom);
                    bMatrix[index]=indigoBondOrder(bond);
                }
            }

            //printf("\n");

            indigoFree(neighborIter);
        }
        indigoFree(atomIter);
    }

    indigoFree(reactantIter);

    productIter = indigoIterateProducts(reaction);

    while((product = indigoNext(productIter)))
    {
        atomIter = indigoIterateAtoms(product);
        while((atom = indigoNext(atomIter)))
        {
            mappingNumber = indigoGetAtomMappingNumber(reaction,atom);
            //printf("%i\n",mappingNumber);
            //indigoGetExplicitValence(atom, &valence);
            //printf("%i\n",valence);
            //eMatrix[mp(mappingNumber, mappingNumber, totalAtoms)]=valence;

            neighborIter = indigoIterateNeighbors(atom);

            while((atom = indigoNext(neighborIter)))
            {
                targetMappingNumber = indigoGetAtomMappingNumber(reaction,atom);

                index = mp(mappingNumber, targetMappingNumber, totalAtoms);

                //printf("%i",indigoBondOrder(bond));
                if(eMatrix[index]==0)
                {
                    bond=indigoBond(atom);
                    eMatrix[index]=indigoBondOrder(bond);
                }
            }

            //printf("\n");

            indigoFree(neighborIter);
        }
        indigoFree(atomIter);
    }

    indigoFree(productIter);

    for(int i=0; i<(totalAtoms*totalAtoms); i++)
    {
        bondMakeBreak=eMatrix[i]-bMatrix[i];

        //rMatrix[i]=bondMakeBreak;

        if(bondMakeBreak>0)
        {
            if(bMatrix[i]>0)
                printf("change: %i to %i\n",(i / totalAtoms)+1,(i % totalAtoms)+1);
            else
                printf("make: %i to %i\n",(i / totalAtoms)+1,(i % totalAtoms)+1);
        }
        else if(bondMakeBreak<0)
        {
            if(bMatrix[i]>1)
                printf("change: %i to %i\n",(i / totalAtoms)+1,(i % totalAtoms)+1);
            else
                printf("break: %i to %i\n",(i / totalAtoms)+1,(i % totalAtoms)+1);
        }
    }

    //printf("2:%s\n",indigoGetLastError());

    indigoSaveRxnfileToFile(reaction,"c:/temp/testrxn_mapped.rxn");

    //printf("3:%s\n",indigoGetLastError());

    indigoFree(reaction);
    indigoReleaseSessionId(session);
    free(bMatrix);
    free(eMatrix);
    //free(rMatrix);

    return 0;
}

At the moment it has the following shortcomings:

50% too much memory consumption, since only half of the R-Matrix would suffice
Each bond is found twice (Atom1 to Atom2 and Atom2 to Atom1)
The make/break/change info is not added to the mapped reaction file

Have fun!

Putting things into perspective

2011-05-23T13:54:00.001+02:00

How to do it the wrong way:

Radiation in Japan

I am no physicist, but this is so straightforward ignorant of factors like long term exposure effects, ingestion of radionuclides and non-uniform contamination that it leaves me speechless.

Dear silicon valley executives with geiger counters, leave them at home. Please.

JSDraw available again

2011-05-09T13:19:00.001+02:00

Apprently, the HTML/JavaScript chemical editor JSDraw is available again:

JSDraw

As being haunted by 'issues' with plugin type editors based on Java or ActiveX for years, it's nice to see that browser-native editing/depiction of chemical structures is gaining more and more momentum.

The other players I know of are:

ChemWriter
Ketcher
Sketcher 2D

Java in-memory compression

2011-04-29T12:29:00.012+02:00

Sometimes it would be nice to have a means to compress seldom used objects in memory to time for space (since compression/decompression takes a bit of overhead time). Especially in chemoinformatics applications where objects often carry substantial compactible information, e.g. textual representations of structures, the compression ratio is quite noticable.

Sure, one could do it for every variable individually, but how about compressing whole Objects using the standard serializing mechanism?

If you use WEKA, you have the SerializedObject.

If not - enter the CompressedReference:


import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class CompressedReference<T extends Serializable> implements Serializable {

private static final long serialVersionUID = 7967994340450625830L;

private byte[] theCompressedReferent = null;

public CompressedReference(T referent) {
try {
compress(referent);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

public int size() {
return theCompressedReferent.length;
}

public  T get() {
try {
return decompress();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}

private  void compress(T referent) throws IOException {

ByteArrayOutputStream bos = new ByteArrayOutputStream();
GZIPOutputStream zos = new GZIPOutputStream(bos);
ObjectOutputStream ous = new ObjectOutputStream(zos);

ous.writeObject(referent);

zos.finish();

bos.flush();

theCompressedReferent = bos.toByteArray();

bos.close();
}

@SuppressWarnings("unchecked")
private  T decompress() throws IOException, ClassNotFoundException {
T tmpObject = null;
ByteArrayInputStream bis = new ByteArrayInputStream(theCompressedReferent);
GZIPInputStream zis = new GZIPInputStream(bis);
ObjectInputStream ois = new ObjectInputStream(zis);
tmpObject = (T) ois.readObject();

ois.close();

return tmpObject;
}
}

A quick test shows 528 byte size for a String of 250 characters (since Unicode needs two bytes per char) and 64 bytes after compression, a ratio of about 8:1. The only requirement is that the Object stored in the reference has to implement Serializable.

And yes, I have to clear those TODO reminders... ;-)

Update
The compression ratio for a CDK Molecule representation of 2-Fluoronaphthalene is about 3:1.
The compression ratio for the V2000 mofile of 2-Fluoronaphthalene is about 7:1.

pgchem::tigress sets new world record!

2010-11-26T13:37:00.000+01:00

The IPB now has successfully loaded ~28 million structures from pubchem into a pgchem::tigress / PostgreSQL database.

This is the largest installation I know of and therefore sets a new world record - for pgchem. :-)

Development of pgchem::tigress 2.0 has started

2010-11-12T14:59:00.008+01:00

I finally managed to get OpenBabel 2.3.0 and PostgreSQL 9.x to talk to each other on Windows. This only works when building with the MinGW Makefiles, not with MSYS, as PostgreSQL does not like mixing with the MSYS runtime libraries. And still a static build of libopenbabel and libinchi is required.

The overall good news is that the Makefile could be simplified a lot with help from Steffen Neumann from the Leibniz Institute of Plant Biochemistry and all current functions are still working with OB 2.3. The build process on Windows is still a bitch compared to building on Posix systems, though.

The roadmap for 2.0:

A lot of code needs cleanup, ~~especially the make_molecule() function, which is unreadable and error prone~~. Mr. Neumann has also identified various possible memory leaks that need to be fixed
~~Allow 2D coordinate generation for Molfile output~~
~~Using OBQuery for non-SMARTS substructure searching~~
~~Add Spectrophore(TM) output~~
~~Add support for exporting Andrew Dalke's chemfp formats (at least FPS)~~
~~Move SVG output from Dingo to OB, having one library less to care for~~
Add the fingerprint linear optimizer in a generally usable implementation
See if the new stereochemistry inside OB fixes various issues that were reported
~~Basic reaction support~~
Win64 builds ?

This will take some time, so don't hold your breath. Since there are already some bugs reported for OB 2.3.0 I guess I'll have to wait for those being fixed before a production release of pgchem::tigress 2.0 becomes available.

GCC6 Day Three (09.11)

2010-11-09T14:23:00.002+01:00

So, I rediscovered my notes from day three. Unfortunately there are only two lectures I can really write something about:

Noel O'Boyles lecture about the in-silico design of polymers with optimal properties for organic solar cells. As far as I understood this, the predicted efficency for organic solar cells is about 13 %. Experimentally 6 % have been reached yet. By using OpenBabel together with cclib and Gaussian, synthetically accessible polymers with 11 % predicted efficiency could be designed.
Roger Sayle from NextMove showed his recent work on a chemistry aware spell checker, based on the observation, that the current problems with chemical text mining do not come from poor OCR or poor name2structure conversion, but mainly from bad input because of typos etc. in the source texts and common spell checking software cannot cope with this.

GCC6 Day Two (08.11)

2010-11-08T16:46:00.017+01:00

So, I rediscovered my notes from day two. Unfortunately there are only two lectures I can really write something about:

The WizePairZ algorithm was quite interesting. It aims to solve the problem that often unwanted properties of drug candidates scale in correlation with wanted properties. The idea behind WizePairZ is to automatically find transformations from one moelcule to another that reduce unwanted properties (ideally while improving wanted properties). Then such transformations are applied to molecules which are close to the boundary between 'unwanted' and 'wanted' space in order to push them over the boundary.
The in-silico prediction of Phototocicity aims to find molecular descriptors and associated models that allow to predict the potential phototoxicity of substances, especially their UV absorption between 290 - 450 nm wavelength where the human skin doesn't filter anymore and the ozone layer doesn't filter yet.

GCC6 Day One (07.11)

2010-11-08T06:57:00.003+01:00

Arrived 14:00 in Goslar

What's new in Knime presentation:
Mainly usability improvements, like conditional paths through workflows and annotations. This is really getting somewhere. Release planned for December 6th 2010

My presentation:
I was not boohed from the stage

MOSGrid presentation:
It's a molecular simulation grid driven by a consortium of academia and industry, lead by the University of Cologne

The planned usage is, that you can specify a task through a web interface in MSML (molecular simulation markup laguage), which describes a task for the grid, then the MSML is translated into input specific to a program (e.g. Gaussian).
The program is then run on the grid and the program specific output is translated again into MSML and returned to the user.

Currently

Gaussian
Turbomole

are supported.

Tasks can be chained into workflows. Lots of expertise and manpower, and if buerocracy doesn't step in the way, this could work and is worth a look. Unfortunately, the licensing of the used programs in a grid seems to be an unsolved legal problem.

Demonstration of MOSGrid:
A basic web interface (to Gaussian I guess) was shown and how to run quantum calculations on the grid from there. The interface is designed for humans. Currently there are no plans to offer this as a service to machines. Licensinsg issues of using commercial software in a grid must be ironed out to make it widely available, but beta testing will start in December to a limited number of users.

GCC6 in Goslar

2010-11-05T09:05:00.000+01:00

I'll be at the 6th German Conference on Chemoinformatics in Goslar from November 7-9.

The SMARTSViewer

2010-11-04T14:18:00.001+01:00

This is a nifty service for visualizing SMARTS patterns from the University of Hamburg:

SMARTSViewer

Probably useful when your SMARTS pattern does not match what you think it should...

Nasty thoughts

2010-10-27T15:22:00.002+02:00

Typically reading the FDA Warning Letters is pretty dry stuff, but this looks like a fun product.

"The effects of ... will promote a thrilling energy, more stamina, constant readiness, nasty thoughts, prolonged arousal, feelings of well being, romantic and sensual experience."

"Nasty thoughts" - those guys from Natural Wellness, LLC really should try Mefloquine...

Searching Structures in ChemSpider from the Android Browser...

2010-08-11T11:05:00.001+02:00

...is now implemented as empirical formula search.

Not what was ultimately desired, but at least it works.

Searching Structures in ChemSpider from the Android Browser...

2010-08-06T10:46:00.001+02:00

...does not work at the moment.

Having, with the friendly help of ChemSpider support, found out how to use their web API, I tried to call it from the android browser.

The HTML form works at least in Firefox, IE 6.x and Chrome, but from the Android browser it either opens the ChemSpider start page - or sometimes only a blank page.

It seems that I'll have to monitor what the Android browser really sends to ChemSpider with Wireshark to find out what the problem is.

Thesis abstract

2010-07-23T10:54:00.001+02:00

The storage and retrieval of chemical graphical datatypes such as structures andreactions in relational database systems is a common technique used in academia andindustry alike. Due to the computationally intensive algorithms used for (sub)graphisomorphismdetection, such systems commonly use faster screening mechanisms inorder to reduce the set of potentional match positives before applying aforementionedalgorithms.

Widely used screening mechanisms are based on numerical and binary vectors, calledfingerprints, with a clear dominance of binary fingerprints due to the raw speedadvantage of bitwise operations and compactness in storage. The two most commonlyused types of binary fingerprints are path-generated and substructure-generated, bothof which have specific shortcomings, especially blind spots.

To overcome this shortcomings, the Pgchem::Tigress chemistry extension to thePostgreSQL object-relational database management system uses a hybrid binaryfingerprint, consisting of an invariant path-generated part and an substructure-generatedpart which is externally configurable through a dictionary of substructure patterns.

This thesis presents a novel approach of using dynamic discrete optimization to find anoptimized dictionary configuration for the substructure-generated part of the fingerprintfor arbitrary sets of structural data.

By means of applying the method developed in this thesis, the computational powerneccessary to run a chemical information system can be reduced by 42 percent onaverage. By improving the query throughput, upgrading the server hardware to thenext level of computational power can be avoided and thus opportunity revenues ofthe operating costs are realized.

Update: It is now readable online.

Thesis defence

2010-07-19T13:26:00.002+02:00

The date of my defence is 21.07.2010, 10:00-11:15 UTC+2.

If somebody's willing to cross their fingers for me, it'll be appreciated. :-)

ChemSpider Web API, anyone?

2010-07-09T11:34:00.006+02:00

I'm trying to do a structure search using the published ChemSpider Web API: http://www.chemspider.com/WebAPI.aspx

I thought it would be sufficient to get the form they provide, put a molfile into the designated area, add an submit button and it works.

The curious case of the infinite canvas

2010-07-05T15:14:00.004+02:00

After some fruitless attempts to extend the usable canvas of DCE ChemPad with some kind of self-made view port, in order to allow drawing of larger-than-screen structures, I think I now found a more painless way.

Simply wrapping the DCE Control in a HorizontalScrollView was not enough, because the ScrollView intercepts too much MotionEvents, but after subclassing HorizontalScrollView and overriding onInterceptTouchEvent(), it seems to work as expected. The code needs some more polishing though to call it working though.

As you can see on the screenshot, the structure now can be scrolled left and right as needed to draw more than would fit on the screen. The thin line on the bottom separates the drawing from the scrolling sensitive area, because otherwise the whole screen would be scroll sensitive and drawing while scrolling is not very precise. The screenshot is from a NexusOne with Android 2.2.

DCE ChemPad Update 1.1

2010-06-02T12:06:00.003+02:00

A few bugs have been fixed (sorry, no autolayout yet :-)), all controls have been moved into the menu and no titlebar anymore to save screen estate. The vibration feedback for fusing atoms works now as expected and you can send the molfile via the phone's messaging systems. Since it seems that attachment handling is ~~broken~~ not fully implemented on Android, the molfile is embedded as text in the message itself. A localized help function was added to the application.

DCE ChemPad 1.0 is out in the wild

2010-05-26T11:11:00.003+02:00

DCE ChemPad is released for free on the Android Market!

It shows the capabilities of the DCE Chemistry Editor Control to add a chemical editor to arbitrary Android applications.

It was tested on the HTC Nexus One, ACER Liquid S 100, Motorola Milestone/Droid and the Emulator and should work with all Android versions >= 1.5. It does reportedly not work on the Motorola Cliq.

If you have acceess to the Market, please try it and tell me what you think...

Ah, screenshots:

Aspirin for Android

2010-05-07T22:43:00.001+02:00

Solving the 'big finger vs. tiny screen' problem

2010-04-22T21:35:00.004+02:00

Learn how to make custom View controls - check
Learn to make compound controls - check
Learn how to paint on the canvas - check
Design a effective 2D rendering pipeline for undirected graphs- check
Remember basic planar trigonometry - check
Design a fuzzy lock-on selection method for the touchscreen - check
Design a robust backing model - check
Design an effective UI for the touchscreen - check
Implement all the nasty details - check

NexusOne: Big finger vs. tiny screen

2010-04-14T10:37:00.004+02:00

I now have a NexusOne at hand.

And while it is much faster than the Emulator, a very profane problem has come up. It is impossible to precisely draw a chemical structure with an editor designed for the mouse! It is just to sensitive to use it with a finger on a tiny screen. And a pen won't work with a capacitive touchscreen...

Maybe i have to take an intensive look at the android.graphics package.

JavaScript molecule editor roundup

2010-04-01T13:39:00.003+02:00

Next to JsDraw which had it's 1.0 release recently, I've found two more pure JavaScript molecule editors:

WebCME which has a lot of features, notably a large library of molecules but is quite painful to use. This is because it's developers have chosen a system of 'select two atoms, add bond, deselect them, select another two atoms, add bond, oh wrong one, delete bond, add correct bond...' for drawing.

The ChemDoodle web components on the other hand have a totally minimalistic, yet powerful UI. Atoms are drawn by mouseclick, bonds drawn by mouse drag. Atom types can be changed via keyboard, bond types by mouseclick and delete is done by the Backspace or Delete key.

Unfortunately none of the three works on mobile browsers. Either they don't work at all or only by half.

In contrast, the jsMolEditor does work even in Android WebViews, but seems to be not under development anymore. I suspect that it was abandoned in favour of JsDraw.

What a pity. Having a (even simple) but working JavaScript molecule editor, that works in Smartphone WebViews would open a whole new world of applications for those devices. They now have the computational power to handle chemical data, but who wants to enter SMILES strings by hand...

pgchem::tigress 1.2 is out

2010-04-01T10:49:00.000+02:00

http://pgfoundry.org/projects/pgchem/

Built and tested against PostgreSQL 8.4.2 with OpenBabel 2.2.3 on XP 32 bit, Windows 7 64 bit and Ubuntu 8.04 LTS 32 bit.

MACCS166 binary fingerprints.

Dice and Tversky similarity.

Small bug fixes.

How to run mx on Android

2010-03-16T10:47:00.001+01:00

mx runs on Android!

To make it work, you need to build from sources and remove all references to javax.swing (which doesn't seem to break the rest of the code btw.), since Android does not contain AWT or Swing.

Then repackage the jar and it can be used in Android applications.

Chemistry on the smartphone, yay. :-)

Changing the public API between minor versions of PostgreSQL

2010-03-10T10:32:00.000+01:00

I hate when they do this.

Between 8.3 and 8.4, the API for CREATE OPERATOR CLASS has changed. Now the RECHECK flag is obsolete, letting the index dynamically decide if it is lossy or not.

While this in itself is an improvement, it generates an incompatibility between GiST C code and scripts written for 8.3 and 8.4. Fortunately, the fix seems to be an easy one...

Wrapping native libraries with JNA: Dingo

2010-03-03T13:55:00.002+01:00

Even with the advent of pure Java chemoinformatics toolkits like MX or the CDK, there is a lot of interesting native code floating around on the net. Unfortunately, wrapping native code with JNI is no real fun.

JNA comes to the rescue. It does all the neccessary loading and marshalling stuff dynamically in the background for you. All you need is a declaration of the native interface, the rest is magic.

Here's an incomplete but working example for Dingo 1.0:

package your_package_here;
import com.sun.jna.Native;
public class NativeDingoWrapper {

static {
Native.register("dingo");
}

public static native int dingoSetOutputFormat(String anOutputFormat);
public static native int dingoSetColoring(int aColoringFlag);
public static native int dingoSetHighlightColorEnabled(int aHighlightFlag);
public static native int dingoSetHighlightThicknessEnabled(int aHighlightThicknessFlag);
public static native int dingoSetStereoOldStyle(int aStereoFlag);
public static native int dingoSetImageSize(int aWidth, int aHeight);
public static native int dingoLoadMolFromString(String aMol);
public static native int dingoLoadMolFromFile(String aFile);
public static native int dingoSetOutputFile(String anOutputFile);
public static native int dingoRender();
public static native int dingoMoleculeIsEmpty();
}

And that's it.

The only drawback of JNA is that it needs a glue DLL specific to the operating system, so theoretically it is more platform limited than JNI.

But since "JNA has been built and tested on OSX (ppc, x86, x86_64), linux (x86, amd64), FreeBSD/OpenBSD (x86, amd64), Solaris (x86, amd64, sparc, sparcv9) and Windows (x86, amd64). It has also been built for windows/mobile and Linux/ppc64, although those platforms are not included in the distribution." this is a quite limited limitation for most cases.

I have successfully wrapped Dingo and Barsoi with JNA and up to now it just works as advertised.

Chemoinformatics in the browser: Fingerprint similarity calculation

2010-02-15T13:12:00.004+01:00

Well there are other things that can be done in JavaScript beyond substructure search. For example, Tanimoto binary fingerprint similarity calculation needs just two short functions:

function popcount(b) {
var c, bi3b = 0xE994;
     c = 3 & (bi3b >> ((b << 1) & 14));
     c += 3 & (bi3b >> ((b >> 2) & 14));
     c += 3 & (bi3b >> ((b >> 5) & 6));
return c;
}

function tanimoto(fp1, fp2) {
var a=0;
var b=0;
var c=0;

for (var i=fp1.length-1; i>=0; i--) {
    var block_fp1=fp1[i];
    var block_fp2=fp2[i];
    a += popcount(block_fp1);
    b += popcount(block_fp2);
    c += popcount(block_fp1 & block_fp2);
}
return c/(a+b-c);
}

The fingerprints have to be converted into JavaScript arrays of equal length containing signed numbers:

onclick="alert(tanimoto(new Array('1','-1073741825'),new Array('3','2147483647')));"

0.9

Chemoinformatics in the browser: Firefox catches up

2010-02-12T09:23:00.002+01:00

>That's a big difference. Which version of Firefox? If 3.5, have you tried 3.6?

Yes, today. Chrome 4 is not faster than Chrome 3 but Firefox 3.6 now allows jobs of about 50 structures.

Browser	max. job size
Chrome 3.x	100
Chrome 4.x	100
Firefox 3.5.x	25
Firefox 3.6.x	50
IE 6.x	5
IE 8.x	10

Those batch sizes allow for script execution times of about 1 second. The idea behind this is, that this does not interfere with other scripts on a page if the job is running embedded, e.g. in an invisible iframe.

If the page is dedicated, much larger jobs might be possible up to the limit of the browser that triggers the 'A script is not responding' error message.

Update: IE 8.x is twice as fast as IE 6.x, but still slow compared to the competitiors.

Chemoinformatics in the browser: Chrome finishes first

2010-02-11T09:31:00.012+01:00

While developing my little demo in the previous article, I found that different browsers could handle different job sizes depending how fast their JavaScript engines are.

Chrome 3.x finishes first before Firefox 3.x and ye olde IE 6.x is almost unusably slow for substructure searching with JavaScript.

The possible job sizes are:

Browser	max. job size
Chrome 3.x	100
Firefox 3.x	25
IE 6.x	5

Thus, the server sizes jobs according to the user-agent header sent:

if (uatype.find('Firefox/3') != -1):
timeout = 500
maxsize = 25
elif (uatype.find('Chrome/3') != -1):
timeout = 200
maxsize = 100
elif (uatype.find('MSIE') != -1):
maxsize = 5
timeout = 1000
else:
return

While I knew that Chrome's JavaScript engine is fast, I didn't expect it to be that dominant.

Browsers of the world: Map! Reduce! Map! Reduce!

2010-02-08T14:21:00.016+01:00

This article about the idea of collaborative map/reduce in the browser and this one on Depth-First gave me the idea to try something other than distributed word counting: distributed substructure matching.

The server was quickly written in Python, the backend in this case is Postgresql with a table holding the structures as V2000 molfiles in plain text format. No magic so far.

Here's the code of the server.

More interesting might be, how the substructure matching itself is done with 100% JavaScript. Thanks to JSDraw a pure JavaScript structure editor, which on closer inspection has some more interesting tricks up it's sleeve, notably a substructure matching capability, this is doable now.

The server schedules a job of maxsize random molecules from the database and constructs a page containing those molecules as molfiles. After the page has completely loaded in the browser, the matching is done and the page is posted back to the server which parses the result. Once manually started by opening http://:8080/get, the pages keep reloading automatically by means of a meta http-equiv="refresh" in the result page.

Of course, the server is very basic. It notably lacks keeping track of the results and housekeeping to restart broken jobs and uses a hardcoded substructure as search argument.

But it can be done.