Archive

Archive for the ‘Programming’ Category

Django and SQLAlchemy session transaction errors

Issue experienced here is related to deploying a django application with SQLAlchemy as the ORM not the in-built Django ORM. Its my first time to deploy Django with a different ORM, in this case SQL Alchemy. The error faced here didn’t happen when running on the django dev server but in Multi-threaded environment (Using Gunicorn). My production environment is comprised of Gunicorn and Nginx on an Amazon micro instance running Ubuntu AMI. Reason for using SQLAlchemy is shared common lib between django web application, RabbitMQ workers and other python tools

Errors

The transaction is inactive due to a rollback in a subtransaction. On subsequent requests,
This Session’s transaction has been rolled back due to a previous exception during flush.

How I am using the session?

GLOBAL_ENGINE = get_engine()
SESSION_MAKER = scoped_session(sessionmaker(bind=GLOBAL_ENGINE))

def get_session():
    return SESSION_MAKER()

Why the error?

Quote from the SQLAlchemy site:-
This is an error that occurs when a flush() raises an exception, rolls back the transaction, but further commands upon the Session are called without an explicit call to rollback() or close().

How to fix:

Create a middleware class that commits and closes the session at the end of the request and on any exception
My middle ware class

class SqlAlchemySessionMiddleWare(object):    
    def process_response(self, request, response): 
        try: 
            session = Repository.get_session()
            session.commit() 
            session.close() 
        except Exception, err: 
            pass
        return response 

    def process_exception(self, request, exception): 
        try: 
            session = Repository.get_session()
            session.rollback() 
            session.close() 
        except Exception, err: 
            pass

Register the middleware class in settings

MIDDLEWARE_CLASSES = (
    ...
    'MiddleWare.SqlAlchemySessionMiddleWare'
)

Restart the web server.

Categories: Programming Tags: ,

Redgate Saves Me Again:- Source of Memory Leak Found

It won’t be the first, and for sure not the last. Redgate tool belt has a set of tools which come in handy when your at the edge of giving up. I have used the database refactor tool, to make nightmares go away, and schema migrations horrors are in the past. Have read of memory leaks on blogs but had yet faced the challenge of tracking one.

Over the last month, I have seen my windows service which starts and runs threads to handle different jobs increase in memory till 900MB in a period of two weeks. Restarting the service fixed the issue but i had failed to find the reason why.

I should say, I tried a couple of solutions including live debugging on the production server and my dev machine. Each day I tried a “trick” but zilch. Until it hit me, Redgate has a performance and memory profiler. I last used Redgate Performance profiler to hunt down beasts ages ago, in any case the tools are just click, wait, results and aaaaaahhhh ok, fix that. Performance profiler didn’t get me much. Ran the memory profiler for 3 hours taking snapshots every 30 minutes. Luckily by just looking at the results it was obvious who the culprit was. Googled LocalDataStoreElement, and the results where in my favor. In the code I was storing thread specific data using Thread.AllocateDataSlot etc, and from
http://support.microsoft.com/kb/2540745, the issue occurs because the Thread.AllocateDataSlot method in the .NET Framework 4 creates a memory leak. Sigh!!!. Now how to fix it?   Below I quote from the microsoft site:-

A supported hotfix is now available from Microsoft. However, it is intended to correct only the problem that this article describes. Apply it only to systems that are experiencing this specific problem.

To resolve this problem, contact Microsoft Customer Support Services to obtain the hotfix. For a complete list of Microsoft Customer Support Services telephone numbers and information about support costs, visit the following Microsoft website:

Note In special cases, charges that are ordinarily incurred for support calls may be canceled if a Microsoft Support Professional determines that a specific update will resolve your problem. The usual support costs will apply to additional support questions and issues that do not qualify for the specific update in question.

Great that I have the code and I can apply the recommended fix for switching from Thread.AllocateDataSlot to ThreadLocal<T>. For minimal changes to code, I made use of ThreadLocal<IDictionary<string, object>>.  I do appreciate the memory profiler tool for working the charms, if it wasn’t for the scary price tag, I would purchase the tool after the trial ends.

Categories: Programming Tags:

MSBuild And MVC Area Projects using Autofac as IoC

April 22, 2012 2 comments

I have a Web Project Folder structure MainMVCProject\Areas\MyArea in one of my projects. MyArea is a another MVC project. While setting up an MSBuild project, I realised that on building the main project using the task below, the contents of the child area MyArea where being copied to the parent MainMVCProject.

<MSBuild Projects=”$(WebFolder)\MainMVCProject.csproj”
Targets=”ResolveReferences;_CopyWebApplication”
Properties=”WebProjectOutputDir=$(DeployFolder)\www;
OutDir=$(DeployFolder)\www\bin\;Configuration=$(Configuration)” />

At first it was not obvious why, but managed to figure it out. The cause was because I had added a reference of MyArea project to MainMVCProject. The reason in the first place of adding that reference was to simplify the registration of controllers in the MyArea child Area project assembly using Autofac. Removing the reference fixed the MSBuild output but still needed to register the controllers with the Autofac Container.

In my solution I use a plugin architecture used in Umbraco, details here. In combination with a TypeFinder, I was already using a DependencyRegistrar for registering types to the Autofac Container. So I just added an implementation of IDependencyRegistrar in the MyArea child project which registers the controllers in that assembly. In Global.asax Application start method I find all types that do implement IDependencyRegistrar and execute the register method passing the container.

This overall fixed resolving controllers at run-time.

Categories: Programming Tags: ,

AES 128bit Cross Platform (Java and C#) Encryption Compatibility

September 21, 2011 73 comments

It seems quite a minor issue but doing cross platform encryption and decryption specifically AES was a bit a challenge for me. When you  get it working it just seems like no big change to the original code you where working on but you wasted fruitless hours debugging. Just sharing so that another person doesn’t have to waste time wondering why same settings produce a different cipher and going through a number of unhelpful posts where it ” works on my machine”.

I initially used ISO 10126 padding, but that seemed to produce a different cipher in C# and Java. I decided to go with no padding which leads me to the  cryptographic exception: Length of the data to decrypt is invalid. After some head banging I settled with PKCS7

Same encoding of strings has to be used in java and c#, other wise you will end up having different ciphers in java and c#.

The java code below uses a base64 util class from android SDK but you can replace it like with one from apache commons

C# encryption utility

public RijndaelManaged GetRijndaelManaged(String secretKey)
        {
            var keyBytes = new byte[16];
            var secretKeyBytes = Encoding.UTF8.GetBytes(secretKey);
            Array.Copy(secretKeyBytes, keyBytes, Math.Min(keyBytes.Length, secretKeyBytes.Length));
            return new RijndaelManaged
            {
                Mode = CipherMode.CBC,
                Padding = PaddingMode.PKCS7,
                KeySize = 128,
                BlockSize = 128,
                Key = keyBytes,
                IV = keyBytes
            };
        }

        public byte[] Encrypt(byte[] plainBytes, RijndaelManaged rijndaelManaged)
        {
            return rijndaelManaged.CreateEncryptor()
                .TransformFinalBlock(plainBytes, 0, plainBytes.Length);
        }

        public byte[] Decrypt(byte[] encryptedData, RijndaelManaged rijndaelManaged)
        {
            return rijndaelManaged.CreateDecryptor()
                .TransformFinalBlock(encryptedData, 0, encryptedData.Length);
        }

        /// <summary>
        /// Encrypts plaintext using AES 128bit key and a Chain Block Cipher and returns a base64 encoded string
        /// </summary>
        /// <param name="plainText">Plain text to encrypt</param>
        /// <param name="key">Secret key</param>
        /// <returns>Base64 encoded string</returns>
        public String Encrypt(String plainText, String key)
        {
            var plainBytes = Encoding.UTF8.GetBytes(plainText);
            return Convert.ToBase64String(Encrypt(plainBytes, GetRijndaelManaged(key)));
        }

        /// <summary>
        /// Decrypts a base64 encoded string using the given key (AES 128bit key and a Chain Block Cipher)
        /// </summary>
        /// <param name="encryptedText">Base64 Encoded String</param>
        /// <param name="key">Secret Key</param>
        /// <returns>Decrypted String</returns>
        public String Decrypt(String encryptedText, String key)
        {
            var encryptedBytes = Convert.FromBase64String(encryptedText);
            return Encoding.UTF8.GetString(Decrypt(encryptedBytes, GetRijndaelManaged(key)));
        }

Java Encryption Utility

private final String characterEncoding = "UTF-8";
	private final String cipherTransformation = "AES/CBC/PKCS5Padding";
	private final String aesEncryptionAlgorithm = "AES";

	public  byte[] decrypt(byte[] cipherText, byte[] key, byte [] initialVector) throws NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException, InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException
	{
		Cipher cipher = Cipher.getInstance(cipherTransformation);
		SecretKeySpec secretKeySpecy = new SecretKeySpec(key, aesEncryptionAlgorithm);
		IvParameterSpec ivParameterSpec = new IvParameterSpec(initialVector);
		cipher.init(Cipher.DECRYPT_MODE, secretKeySpecy, ivParameterSpec);
		cipherText = cipher.doFinal(cipherText);
		return cipherText;
	}

	public byte[] encrypt(byte[] plainText, byte[] key, byte [] initialVector) throws NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException, InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException
	{
		Cipher cipher = Cipher.getInstance(cipherTransformation);
		SecretKeySpec secretKeySpec = new SecretKeySpec(key, aesEncryptionAlgorithm);
		IvParameterSpec ivParameterSpec = new IvParameterSpec(initialVector);
		cipher.init(Cipher.ENCRYPT_MODE, secretKeySpec, ivParameterSpec);
		plainText = cipher.doFinal(plainText);
		return plainText;
	}

	private byte[] getKeyBytes(String key) throws UnsupportedEncodingException{
		byte[] keyBytes= new byte[16];
		byte[] parameterKeyBytes= key.getBytes(characterEncoding);
		System.arraycopy(parameterKeyBytes, 0, keyBytes, 0, Math.min(parameterKeyBytes.length, keyBytes.length));
		return keyBytes;
	}

	/// <summary>
	/// Encrypts plaintext using AES 128bit key and a Chain Block Cipher and returns a base64 encoded string
	/// </summary>
	/// <param name="plainText">Plain text to encrypt</param>
	/// <param name="key">Secret key</param>
	/// <returns>Base64 encoded string</returns>
	public String encrypt(String plainText, String key) throws UnsupportedEncodingException, InvalidKeyException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException{
		byte[] plainTextbytes = plainText.getBytes(characterEncoding);
		byte[] keyBytes = getKeyBytes(key);
		return Base64.encodeToString(encrypt(plainTextbytes,keyBytes, keyBytes), Base64.DEFAULT);
	}

	/// <summary>
	/// Decrypts a base64 encoded string using the given key (AES 128bit key and a Chain Block Cipher)
	/// </summary>
	/// <param name="encryptedText">Base64 Encoded String</param>
	/// <param name="key">Secret Key</param>
	/// <returns>Decrypted String</returns>
	public String decrypt(String encryptedText, String key) throws KeyException, GeneralSecurityException, GeneralSecurityException, InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException, IOException{
		byte[] cipheredBytes = Base64.decode(encryptedText, Base64.DEFAULT);
		byte[] keyBytes = getKeyBytes(key);
		return new String(decrypt(cipheredBytes, keyBytes, keyBytes), characterEncoding);
	}

Hopefully this saves someone time.

Categories: Programming Tags: , ,

My First Android Application

September 15, 2011 1 comment

I was hesitant to learn android development because of time. Last week, a requirement kicked in that needed an android front-end. Target platform was android 2.2.  This is my feel of android development after six days of using android SDK. Actually I had also never used an android phone, so I was amazed how cool android OS  is.I should say mobile development should have been easier like how android development is from the beginning. I have developed in QT, and J2ME, but both dont match up to android development with ease. The great UI controls that come out of the box, the eclipse integrated plugin, the ability to reuse most of the standard java code, its all great.

I did consult the SDK documentation and the hello tutorials during the development. They where of great help. The application pulls the data from a .NET WCF JSON Rest based service. I used the Google Gson for the serialization and de-serialization of json to object types.  On the WCF service I decorated by operation contracts with WebGet and WebInvoke depending on whether am making GET or POST requests. To reuse the forms based authentication, the http connection service takes care of  managing the cookies mainly the forms based authentication cookie to receive and send it on requests.

Coding in android though seams easier than QT or j2me, I got some beating for not reading the documentation. Some of the beating I got:-

  • Embedding more than one root view in the scroll viewer was just throwing exception. I was quite big headed on googling for a solution on this one, because I just couldn’t expect it was the cause.
  • The table layout. This I had to abandon and choose another approach. I needed to replicate a telerik like silverlight gridview with a details view on each row. Was t0o ambitous about it, but abandoned it. Got headaches about the table row layout parameters and scrollview.
  • Adding an activity to the manifest file. Was always forgetting about this step and that led me to debugging. And pretty much I would remember (f***) when the exception is raised from the line where I start the Activity.
  • A point I got so frustrated with exceptions from setting content view. The exceptions just could’t make sense. I don’t know what I f.*up, but when I change layouts, or add new layouts. I tend to get exceptions when setting activity view. It just couldn’t make sense why. so I went to the bin folder and did the mighty SHIFT-DELETE-RETURN. Damn. all references to resources in R.java couldn’t be recognized any more. I was quick to think may be its a build order, so I moved gen above src folder, build project and back. Some magic happened, no more errors even when setting the content view. I still don’t know why for now. But this seems to be the solution I do for now when I get exceptions when setting content view. It seems to work. Till when I become a sniper at android, I will overlook this.

Damn! Hooking my configuration service to android preferences activity has been the easiest way of managing preferences on a mobile app have seen.

Lastly for now. The way resources are managed in android development, I can’t find an execuse for the laziness of harding cording strings in code files. Its damn fast, easy. Android is great. Android development platform is very great. Now I just feel cheated because I was missing a lot for a long time. I must boostrap my N900 with Nitdroid.

Android development is the way mobile development should have been like from the beginning.

Categories: Programming Tags: , ,

Audio FingerPrinting and Matching Using Acoustid Chromaprint on Windows With Python

I needed to generate audio fingerprints for matching/pattern recognition. I should say I tried a couple of approaches like using MFCC and spectrogram matching using Computer Vision techniques but I was not successfully with it. Using acoustid Chromaprint gave positive results. Since I was using windows, I had to compile the latest version using Visual Studio.
Since I my target was to get wav format to work, I didnt need ffmpeg dependency, so I removed it. You will need the avcodec and ffwt libs. The algorithm used for Acoustid for finger print matching is buried in a PostgreSQL user-defined C function which I easily translated to python for use.
Read more…

Illustration of solving sudoku puzzle using Brute Force, Generate and Test, and Arc consistency

There are different ways of solving Sudoku puzzles. Brute force, Generate and Test both perform pretty well for easy puzzles but Arc Consistency beats them all in speed and solving very complex puzzles with less CPU.  Peter Norvig has a robust solution that solve any puzzle .

Brute force, Generate and Test are easy to implement. but what about using Arc Consistency. For adventure I attempted to try out using arc consistency to solve sudoku puzzles. I did manage to implement the concept though my implementation is not as fast as Norvig’s. Read more…