Pablo's blog

A bit of this, a bit of that and a lot about computers

The problem with Lisp

I’m not really going to talk about the problem with Lisp, but only a problem. And I’m not really going to talk about Lisp, but Common Lisp. So this post should be called “A problem with Common Lisp”, but the current title is more catchy, isn’t it?

You are still with me? nice. Because I believe I have something important to say. A problem with Common Lisp is that you can’t make programs with it. That may have been another nice catchy title: “You can’t make programs with Lisp”.

You can make libraries just fine, and package them, and distribute them, and re-use them, but not programs. To give meaning to these statements I’ll have to define what a program is. A program is something that you can run from a shell (like bash, csh, tcsh, etc) by typing its name and pressing enter . Or, if you want another definition, something that you can run from a C program using the function exec.

I’ve asked this question several times on #lisp on Freenode and generally the answer is this: SBCL can generate an runable image. Although it may be nice in some cases: like shipping an alpha version of a program to someone who doesn’t have a clue what Lisp is, it is not a good way to ship a program expecting an OS like Debian or Ubuntu to include them: “25MB for a hello-world? you are crazy”. The size is not the only problem, upgrading the libraries get harder, because you have to upgrade the libraries in the image and not only the modules in the file system that the OS can easily upgrade with tools like aptitude or rpm. They are just monolithic. I can imagine ls and cp being written in Lisp, so I can’t imagine them being huge monolithic images. Any other language that I know doesn’t have this problem, be it Python, C, C++, Java, Objective-C, C#, Haskell, Chicken Scheme, Erlang (this one has its problems but I’ve already solved some of them), Perl, PHP, any other?

Other Lispers claim that you can just ship a library and the user would just run a function on it; like (run-checkers) to play checkers. This has various drawbacks. First, not every user is a Lisper. Most computer users don’t even know what Lisp, so they wouldn’t know how to run a lisp. They also wouldn’t be able to have an icon in their desktops to run the program, they’d have to open a terminal. For people that can do all that, finding the right function to run in one of those so-called programs can be a challenge. My wife wouldn’t have even tried to run any of those programs. Yes, I want to write software for my wife, in Lisp. And last, even when they can do all that, they may still search for an alternative that just puts an icon somewhere and they click it.

Yes, long rant. I hope I could convince you of the need to produce programs. It has been said many times that Lisp needs a killer application, like ejabberd for Erlang or darcs for Haskell. Both awesome applications that get people to approach the language (indeed, what made me learn Erlang was ejabberd). Without being able to make programs, you can’t make killer programs.

After pushing this issue a bit on comp.lang.lisp I’ve got pointed to cl-launch by Alex Mizrahi (I actually had to say that I was going to do something about it for someone to say “don’t re-invent the wheel”). I knew there had to be something out there! I couldn’t believe I was the first experiencing this. There’s hope!.

Unfortunately, not a single cl-launch powered Lisp program seems to be out there. Maybe because there’s no build-tool integration for it anywhere. Let’s see if I can do something about it.

Update (2007-09-05): Two possible solutions I posted: Solving Lisp’s problem: a simplistic solution with make, Another simplistic solution, with SCons.

Comments at the original blog

  1. Pupeno Says:
    Just when I’m about to publish this document, John Q. Splittist covers the same topic on: http://blog.splittist.com/2007/08/25/lisp-faq-how-can-i-generate-a-standalone-executable/

    We have different points of views, which makes it really interesting.

    Note: my article was not inspired by John Q. Splittist’s, I’ve been working on this article for a couple of weeks while I’ve only read John’s today.

    August 26th, 2007 at 19:16 e

  2. Pupeno Says:

    I’ve submitted this article to Reddit: http://programming.reddit.com/info/2ivn2/comments and it seems most Redditers took it as not wanting to ship or depend on a VM or environment.
    That’s not what I’m saying. What I’m saying is that shipping a monolithic binary, with VM and libraries that doesn’t use the system shared VM and libraries is a problem.
    Also shipping a program that depends on one and only one VM in a world of multiple VMs is a problem.
    And last but not least, I don’t express this as “let’s not use Lisp” but as “let’s fix it” and a “it won’t be hard to fix”.

    August 26th, 2007 at 21:21 e

  3. Pupeno Says:

    One simplistic solution: http://pupeno.com/2007/08/26/solving-lisps-problem-a-simplistic-solution-with-make/

    August 26th, 2007 at 22:25 e

  4. Vitor Says:

    “Also shipping a program that depends on one and only one VM in a world of multiple VMs is a problem.”

    What problems are you REALLY running into?

    You keep making these apples to oranges comparisons with the other languages. They have your odd “VM issue” as well:
    see Python/Jython/Stackless/IronPython or Ruby/JRuby/Ruby.Net or Java/Beanshell/Groovy/Scala. The people that use these languages generally favor and deploy on one VM of the choices they have–why should it be different for Lisp?

    People have made standalone applications with Lisp with little trouble. It takes very little effort to Google the issue and find several answers.

    Feel free to delete this comment like you have deleted others, but realize that you are creating problems that don’t exist.

    August 26th, 2007 at 22:57 e

  5. Anonymous Says:

    “Every other language that I know doesn’t have this problem, be it Python, C, C++, Java, Objective-C, C#, Haskell, Chicken Scheme, Erlang (this one has its problems but I’ve already solved some of them), Perl, PHP, any other?”

    Smalltalk

    August 26th, 2007 at 23:16 e

  6. Pupeno Says:

    Vitor,

    The difference is that in many of those cases there’s actually only one VM/environment. That’s why I believe it is better to just implement a language, like in Python’s case, than have a standard and many implementations. In the case of Java and .Net, the way to build programs is quite easy, due to “java” standard command line flags or .Net standard way to build programs (you can compile with mono and run on the other one and viceversa).
    Most Lisp standalone applications suffer of some problem or another, I have already looked it most free ones I could find and disregard the way they do it because there has to be and there is a better way.

    August 26th, 2007 at 23:38 e

  7. Pupeno Says:

    About Smalltalk, I’d say that actually, due to its heavy use of the “image” concept, it suffers from the same, or worst problem than Lisp itself.
    You may be able to write a program to GNU Smalltalk and ship it sanely, but GNU Smalltalk never ends up looking like a nice implementation for me (and obviously, don’t even dream about cross-implementation programs, but that was never intended, unlike with Lisp).

    August 26th, 2007 at 23:40 e

  8. Juanjo Says:

    Si podés hacer librerías en Common Lisp.. pero no programas.. en que programas usas esas librerías?

    Saludos!!

    August 27th, 2007 at 1:03 e

  9. Pupeno Says:

    Juanjo: hasta ahora, yo diria que en ninguno; en Common Lisp solo hay bibliotecas, no programas.

    August 27th, 2007 at 1:46 e

  10. I had no problems when I installed The Regex Coach http://weitz.de/regex-coach/#install

    August 27th, 2007 at 9:13 e

  11. Pupeno Says:

    About The Regex Coach, that is proprietary software, it probably is a monolithic thing that includes its own Lisp and its own set of every library. Now imagine you have 100 of those programs installed and there’s a bug in a library. You’ll have to upgrade 100 programs with 100 copies of Lisp and of every library instead of just upgrading a library.
    In the free world we like re-usability, so we separate libraries which can easily be upgraded by tools like apt.
    Please, read carefully what I wrote, I already acknowledge the monolithic way of doing things which is actually easy with SBCL, and I already explain why it’s not viable in a OS like Ubuntu or Debian. Please, please read and understand that, I’ve answered this question/complain/issue a hundred times already in different places.

    August 27th, 2007 at 11:00 e

  12. Nathan Says:

    And so we wind up with the current situation, which is that you have the Lisp implementation and lots of separate loadable text files…just like Python, Perl, Ruby, etc. etc. Problem solved. What’s wrong with this again?

    August 27th, 2007 at 14:07 e

  13. Pupeno Says:

    Nathan,
    In Python, Perl or Ruby, you just put a she-bang at the beginning, like #!/usr/bin/env python, and you have a Python program. That’s not possible in Common Lisp. It might be possible in some implementations, in some cases needing customizations that are not widely available.
    If you need further proof of this, check any distribution and you’ll find thousands of programs that use this method made in Python, Perl, Ruby and other interpreted languages. You’ll find absolutely none (well, maybe one) written in Common Lisp.

    August 28th, 2007 at 10:10 e

  14. ken Says:

    This is a nonproblem. Or at least, it’s no worse than any of the other languages you’ve mentioned.

    The Python programs on my system (not written by me) are run via a tiny shell script, which starts the Python VM pointed at some source code.

    I can do exactly the same with Lisp: put some source code in a folder, write a shell script, and point it at them. Presto, a tiny ‘ls’.

    Now, in addition to that, I have other options: I can compile my Lisp code beforehand, I can distribute an image, I can use SBCL’s shebang feature to run a Lisp file directly, and so on. But I can do what all my Python apps do, too.

    For people shipping big serious apps on Windows and Mac, a 25MB app doesn’t look so scary. It *really* doesn’t look scary to the programmer because it means he doesn’t have to implement that stuff himself! 25MB hello world, 26MB Excel.

    Yeah, it might be nice to have a Lisp with compilation facilities more like C. Would I ever need them? Probably not. If you want to help Lisp, go ahead and try to solve this, but don’t blow it out of proportion. We’ve got much bigger fish to fry.

    October 13th, 2007 at 1:49 e

  15. Pupeno Says:

    Ken,

    Here I go again repeating myself. With Python you can put “#!/usr/bin/env python” in top of the source file and you already have a program. You can’t do the same in Common Lisp. For two reason, one is syntactically, and can be solved easily, the other is that Common Lisp is a standard and there are many implementations and you don’t know which implementation you are going to find on a system.
    I’ve said, from the very beginning, that I don’t care about Windows and Mac. In those platforms they have huge programs because they don’t ever share anything. In pretty much the rest of the world we have thousands of small libraries that are independent of each other. So if there’s a bug in one, you only update that one. If we have 100 applications all being 25MB applications, first we need 2500MB just to store, but if there’s a bug in a 2KB library somewhere in there, an aptitude update will have to download 2.5GB instead of 2KB (not counting the huge amount of work that is needed by the packagers to recompile all that). This doesn’t happen in Python or Perl because your program is only your program and the rest is on the system.

    Now, most important, the problem has been solved 90%, so my blog post is more about pointing where the solution is. I posted two other blog posts walking the other 10% (OK, you could say I walked 9%). It was easy, simple and guess what. Nobody cared! Everyone comes here like you, complaining that I complain about a problem without even noticing that I complain about a problem and I immediately point the to the solution. And nobody is using or caring about the existence of a solution: cl-launch. I think that’s because most lispers live in a live of Lisp and they forget about the users, they only care about libraries and not programs. Pick up any Linux distribution and compare the amount of libraries vs the amount of applications and you’ll see it doesn’t make any sense. Specially if you compare it with other languages, even those with ugly syntax or imposing standards or very unpopular (more unpopular than Lisp, that is, with smaller communities) and you’ll see less libraries, but more applications (Haskell, OCaml, Erlang, etc).

    October 13th, 2007 at 13:45 e

  16. TheGZeus Says:

    …couldn’t you just set your favourite Lisp as a dependency to the package?
    Non problem as far as I can see.

    Just keep the code as something.lisp and have a shell script pointing to clisp, cmucl, lush, whatever. I prefer clisp, mainly because it works nicely with SLIME.
    You could install all the implementations in the Ubuntu repositories and not feel a twinge on today’s hard drives.
    They’re not THAT big, and it’s not nearly as bad as the first time a KDE user installs a GNOME program!

    Why package it into a binary if it’s going to be problematic (size or something else) unless you’re not giving up the code?

    December 19th, 2007 at 11:41 e

About these ads

Single Post Navigation

5 thoughts on “The problem with Lisp

  1. Alpheus on said:

    I don’t see this as the biggest problem with Lisp. I’m just starting out, but I’m also determined to learn Lisp…and while I’ll someday be interested in distributing executables, for me the biggest problem I have to come to grips with is the library mess! How the heck do I find libraries that do what I want, and how do I install them?

    I’m inclined to think that if *this* could be solved, the “standalone” executables will be easier to deal with.

    So far, though, I like the language.

  2. Joe Davis on said:

    I haven’t had much trouble doing this with guile, check out my guile-sample project at github.

  3. Uesty on said:

    You say you don’t have this problem with other languages. Can you explain how? How do you compile a Python program to a native executable? How do you compile a Java program to a native executable? I’ve used all 3 languages and they all seem virtually identical in this regard. If anything, Common Lisp is better here because it actually has native compilers, while Python doesn’t (at all, AFAICT) and Java barely does (only fringe compilers that tend to lag the standard and are rarely used).

    • I haven’t touched Common Lisp in quite a while, but the last time I touched this was the situation (I really can’t talk about java). In Python, if I put this on top of a text file:

      #!/usr/bin/env python

      I’m quite sure it’ll run almost everywhere. It would be rare for a system to have Python to fail to run it.

      The same thing was implemented for SBCL but it was implemented poorly and buggy and Lispers were generally against it. If you look for Lisp applications, generally they require you to jump inside a Lisp REPL and run from there, which doesn’t work for user applications or server applications (can’t run it from /etc/init.d/ and so on).

      #!/usr/bin/env ruby
      #!/usr/bin/env perl

      they all pretty much work. There’s one even for Erlang although I would advice against it.

  4. thats have nothing to do with compiling. All you do here is triggering the system provided python intepreter, you don’t create standalone executables. Windows does not come with python preinstalled so your above approach won’t work and of course the above lines says nothing to windows. If you use python 3 then this won’t work even in Linux and Macos unless you expect your user to manually install python 3.

    Python is a great programming language. But there is massive difference between lisp and python. Don’t be fooled by py2exe and py2app , they are both lousy implementations, for me they were pain in the a@@ to use and I never managed to get them working anyway. Probably the closest you can ever get is to use cython (thats cython not cpython) to compile python to c code and then compile it and embed it in a c app.

    Python produces bytecode, lisp produces actual machine code and implementations like ecl compile to c code and then to machine code so they create actual executables.

    Bottom line if you are a big fan of standalone executables done the easy way, stay away from python. Lisp does this much much better.

    But in the end the king of standalone executables is smalltalk. Squeak for example come with all-in-folder, there you can find an environment that works on windows / macos and linux with absolute zero install. You just run the corresponding binary executable for OS you use. So the only thing you need to do in order to make your squeak application standalone and cross platform, is one thing , “absolutely nothing”. Any squeak program will work out of the box with zero effort in any OS system. And because everything is contained in a single folder unlike Java there is no need to install anything.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 375 other followers

%d bloggers like this: