Shell script for updating project in server from local using git

Hi,

This is a trick for lazy people like me. Say, you have a simple project that is being hosted by github or bitbucket. That same project is being hosted in a server too. Then, there is an easy way for porting local changes to server through github or bitbucket.

First you need to add the same remote, either github or bitbucket in your local copy and in your server copy. So that, pushing from local and pulling from server both occurs keeping github or bitbucket in the middle.

Now, the steps involved in syncing the server with local copy after changes is like this:

1. Go to your project folder.
2. Add your changes.
3. Commit these changes.
4. Pull from github or bitbucket.
5. Push to those.
6. Login to your server by using SSH.
6. Go to the project folder in the server.
7. Pull from github or bitbucket.

Quite a few commands need to be executed. Let's add them in a simple script that can do all these.

For that, you need sshpass to insert password while logging into the server with ssh. Now, the script would look like these:

#!/bin/sh
cd "/path/to/your/local/project/" && 
git add . && 
git commit -am"lazy commit message" && 
git pull origin master && 
git push origin master && 
sshpass -p 'your_server_password' ssh username@server_address 'cd /path/to/project/ && git pull origin master'

:)

Find strings in your facebook chat history

Hi,

I've lost a number which my friend gave in facebook chat. But I couldn't remember when he gave it. So, rather than checking all those previous messages manually, I ended up writing a program which would search for a string in facebook chat history.

So, there are three steps involved in this program:

1. Find the id of that conversation
2. Request to the graph api
3. Iterate through the paged responses

To accomplish the first task, send a request to 'user/inbox' node of the graph api. You would find a list of conversations each with it's own id. Find your desired one. It's possible to come up with a program to find it automatically, however, I'm gonna settle for this much for now.

Now, define these three functions:

import requests
from facepy import GraphAPI
import re

def search_in_message(message):
    for m in message['data']:
        try:
            matched_obj = re.match('(\d+)', m['message'], flags=0)
            if matched_obj:
                print m['message']
                print m['created_time']
                print matched_obj.group()
        except KeyError:
            continue
def get_message_from_graph(graph, stream_id, limit, since):
    return graph.get(stream_id + '/comments?limit='+str(limit)+'&since='+str(since))

def get_next_page(next_url):
    return requests.get(next_url).json()

Each methods should be self explanatory. However, we can achieve different results, like a specific number, pattern, string in the messages just by modifying the search_in_message function. I've given it the shape to find numbers in the conversation. The later parts are pretty straightforward:

token = 'your_secret_token'
graph = GraphAPI(token)

paged_messages = get_message_from_graph(graph, 'your_conversation_id', your_limit, your_since_timestamp)

while paged_messages['paging']['next']:
    print paged_messages['paging']['next']
    search_in_message(paged_messages)
    paged_messages = get_next_page(paged_messages['paging']['next'])

Get various information from Facebook Graph API

Finding information from dataset has always fascinated me. And facebook is such a vast reservoir of data, that it's practically a crime for the enthusiasts to leave it untapped.

I've been looking for some way to get different questions answered by facebook.

Like, who has liked my post most? Whose post I liked most? Comments, photos everything can be explored to find these sort of information. There are also a lot of apps for these purposes, but as a developer, you should be able to do it yourself, at least I think so.

I'm going to explain this one: "Who liked your last N number of feeds most?"

In this post, I've explained the way to connect to graph api. Keep in mind that facebook has made the fql obsolete from versions 2.0+. So, being an expert in fql is not going to help much in the future.

Now, facebook has opened up different root nodes in the graph api. They could be found in this link . We are going to use the 'user/feed' node for our purpose. Now, of course facebook is not going to give all data in one request, nor you should expect it to. It returns data in a paged format. There are three types of pagination that facebook uses. Our node uses time based pagination.

Let's look at this code chunk:

import requests
from facepy import GraphAPI
import pprint
from pymongo import MongoClient
from bson.son import SON

client = MongoClient('localhost', 27017)
db = client.likes
like_table = db.like_table

We're going to use mongo a little bit. Due the scope of this post, I'm not going to describe it's installation or usage. This link has very good resources on it.

feeds = graph.get('friends_or_self_id/feed?limit=1')

To find this id, we can simply issue a request in the browser in the address 'http://graph.facebook.com/username'. We could also append parameters like until and since to specify the time period we are concerned about. However, I faced trouble getting them working. So, I've opted for counter instead.

while True:
    try:
        likes = feeds['data'][0]['likes']
        while True:
            try:
                for m in likes['data']:
                    like_table.insert(m)
                likes = requests.get(likes['paging']['next']).json()
            except KeyError:
                print("keyerror in likes pagination")
                break
        if(counter == 30):
            break
        counter = counter + 1
        feeds = requests.get(feeds['paging']['next']).json()
    except KeyError:
        print("keyerror happened")
        break

This shall retrieve the last 30 feeds on the timeline of that user. Then, let's find the most appearance of a particular id. This is where mongo is coming handy.

list = (like_table.aggregate([
    {'$group': {
        '_id': '$id',
        'count': {'$sum': 1}}
    },
    {"$sort": SON([("count", -1), ("_id", -1)])}
]))

for k in list['result']:
    print(k['_id'])
    print(k['count'])

This is just one simple example of the endless possibilities of facebook. I'm a big fan of it just because of it's database. People can practically be 'profiled'.
The code is also hosted in github.

Useful FQL examples

I've been exploring the facebook graph API and fql. FQL seems to be a very powerful tool. Here are some of them which I've been using recently:

1. Finding friends user id by looking up his name:

select name, uid from user where strpos(name, "part_of_your_friends_name") >= 0 and uid in (SELECT uid2 FROM friend WHERE uid1=me())

2. Find the id of a group by its name:

select id, name from profile where name="group_name"

3. Find posts of a user in a group:

SELECT post_id, message, created_time  FROM stream WHERE strpos(field_name,'possible_string') >=0 and  source_id=group_Id and actor_id=user_id  and created_time >=unix_timestamp and created_time < unix_timestamp limit A_NUMBER


//will be continued...

note:
To get the unix timestamp you can use this code in Javascript:

var d = new Date();
d.setMonth(d.getMonth() - 3);
three = Math.round(d.getTime() / 1000); //3 months ago

console.log(three);

Answers to the exercises of the book Natural Language Processing with Python: Chapter 1

from __future__ import division
from nltk.book import *

#author: Anik

Checkout Chapter 2 Excercise Answers

ex 2
print 26**100

ex 3
print ['Monty', 'Python'] * 20

ex 4
print len(text2)
print len(set(text2))

ex 5
romance

ex 6
text2.dispersion_plot(["Elinor", "Marianne", "Edward", "Willoughby"])
Ellenor appears everywhere, and probably Willoughby and Marianne had a relationship

ex 7
print text5.collocations()

ex 8
easy

ex 9
easy

ex 10
my_sent = ["All", "quiet", "on", "the", "western", "front"]
joined =  ' '.join(my_sent)
print joined
print joined.split()

ex 11
phrase1 = ["All", "quiet", "on", "the", "western", "front"]
phrase2 = ["We", "were", "Soldiers"]
print len(phrase1 + phrase2)
print len(phrase1) + len(phrase2)

ex 12
print "Monty Python"[6:12]
print ["Monty", "Python"][1]
second one, tokenized

ex 13
sent = "Flags of our fathers"
print sent.split()
print sent.split()[2][2]

ex 14
sent3 = ['In', 'the', 'beginning', 'God', 'created', 'the', 'heaven', 'and', 'the', 'earth', '.']
indices = [i for i, x in enumerate(sent3) if x == "the"]
print indices

ex 15
bs = [w for w in text5 if w.startswith('b')]
print sorted(bs)

ex 16
print range(10)
print range(10,20)
print range(10, 20, 2)
print range(10, 20, -2)

ex 17
print text9.index('sunset')
print text9[621:644]

ex 18
print sorted(set(sent1+sent2+sent3+sent4+sent5+sent6+sent7+sent8))

ex 19
print len(set([w.lower() for w in text1]))
print len([w.lower() for w in set(text1)])

ex 20
print "ANIK".isupper()
print not "ANIK".islower()
no difference

ex 21
print text2[len(text2)-2:len(text2)]

ex 22
four_letter_words = sorted([w.lower() for w in text5 if len(w) is 4])
print FreqDist(four_letter_words)

ex 23
upper_case_words = [w for w in text6 if w.isupper()]
for word in upper_case_words:
    print word

ex 24
word_list = [w for w in text6 if w.endswith('ize') and 'pt' in w and w.istitle()]
print word_list

ex 25
sent = ['she', 'sells', 'sea', 'shells', 'by', 'the', 'sea', 'shore']
print [w for w in sent if w.startswith('sh')]
print [w for w in sent if len(w) > 4]

ex 26
print (sum([len(w) for w in text1])) / len(text1)

ex 27
def vocab_size(text):
    return len(set([word.lower() for word in text if word.isalpha()]))
print vocab_size(text1)

ex 28
def percent(word, text):
    return 100 * text.count(word) / len(text4)
print str(percent('a', text4)) + '%'

ex 29
too descriptive

Connect to facebook Graph API with python

Yes, this type of posts are all over the world wide web. However, there are thousands of methods and not all of them are going to work for you. I've tried to find the easiest way to interact with Graph API and that's what I'm going to describe, along with the problems I've faced throughout the process. I've been using PyDev in Eclipse.

1. Get python-facebook sdk. this seems to be the official one. Install it with this command

sudo pip install git+git://github.com/pythonforfacebook/facebook-sdk.git

2. If you have opened up eclipse before this process, you are going to get an import error. PyDev doesn't refresh the PYTHONPATH automatically. You have to got to

File > Properties > Edit the launch configuration > Refresh tab > Check refresh all resources

3. Now, you would need an access token to interact with the Graph API. Usually this is done with a browser requesting with app id and secret token and then facebook returns an access token. That's a bit tricky, I was looking for an easier method. However, for testing purpose, facebook also gives access tokens directly.
Go to this link : https://developers.facebook.com/tools/explorer and click on generate access token. It shall ask for the permissions you want to have in your application and then generate an access token for you. However, It's temporary and shouldn't be hard coded in your application forever.

4. Paste the following code in your editor. Among tons of others, I've found this one little and easy.

import facebook
token = 'your secret token'
graph = facebook.GraphAPI(token)
profile = graph.get_object("me")
friends = graph.get_connections("me", "friends")
friend_list = [friend['name'] for friend in friends['data']]
print friend_list

And see the output in console, a nice list of your friends. Remember two points:

1. If it fails to find 'facebook' module, you might need to restart eclipse.
2. If it fails to find Graph API, re-install facebook module. You can remove it by issuing

   pip freeze

   to get the actual python-facebook module name and then

   pip uninstall that-module

    

Oh My Zsh! for Linux Users

If you use linux and do not use zsh then you are missing something you don't want to miss. Actually I'm grateful to my senior for introducing it to me.

Oh My Zshell is a project for extending and beautifying zshell. Follow that link and don't ever look back to any other shells. :)

For installation and usage:

http://railscasts.com/episodes/308-oh-my-zsh
http://www.stevendobbelaere.be/installing-and-configuring-the-oh-my-zsh-shell/

Screenshot of my current shell, using guake and oh-my-zshell :