Local Warming

Yeah, so, Global Warming, erm, I mean, Climate Change, I mean, WGAF….the world is getting warmer and extreme weather events are becoming more common. Anyhow…

I decided to see if there was measurable local warming present; this blog post is to demonstrate a quick and dirty project to illustrate exactly that. NOAA has a lovely website where you can download (FOR FREE!) daily weather data for any zip code in the US. I chose my hometown of Hopewell, VA (ZIP: 23860) as an example starting at 1950-01-01 to the present (2018-05-09 was the latest date).

I read the data into a KNIME workflow, calculated the daily average (TMAX+TMIN)/2, applied an exponential smoother (window size = 365), and then used the R View node to display it with a linear smoother (n=365) to show the trend.

workflow.svg

The plotting code is pretty simple:

library(ggplot2)
ggplot(data=knime.in, aes(x=as.Date(knime.in$”DATE”, “%Y-%m-%d”), y = knime.in$”MA(TAVG)”, group=1)) +
geom_line()+
scale_x_date(date_breaks = “10 year”, date_labels = “%Y”)+
ggtitle(“Hopewell (Zip: 23860) Temperatures 1951-08-15 to 2018-05-09”) +
xlab(“Date”) +
ylab(“Smoothed Average Daily Temperature”) +
geom_smooth(method=lm, n=365, aes(color=”red”))+
theme(legend.position=”none”)

I filtered out the first 365 rows of data, because the exponential smoother broke down at the the beginning of the data.

The result?!? Holy moly, even only over the course of the last ~70 years, there is a clear upward trend!  Local Warming it is!

smoothedTemp

(I haven’t included the workflow itself, because it’s not one of WordPress’s supported media types.)

 

Advertisements
Posted in Data Activisim, Data Mining, Information Visualization, KNIME, Tutorial | Leave a comment

Open Source Orgs

A blog post about Google’s upcoming Summer Of Code showed up in my feed, and it mentioned something about the 212 organizations that had been selected for being mentor organizations. To my shame, I haven’t really paid a lot of attention to the past Summers of Code, other than seeing the (sometimes excellent) results somewhere on the Innertubes.

So, I browsed the list….HOLY MOLY! It’s like being a kid in a candy store with this list. Some of the names were obvious (Fedora, KDE, openSUSE, Apache, and so on), but some of these organizations were completely new to me, like open source health care tools like LibreHealth, and OpenMRS, or Open Food Facts or Open States,  or even telecom tools close to my heart like OpenSIPS. It’s a very cool list and a reminder of just how dynamic, varied, and far-reaching the Open Source community has become in the last decade.

 

 

Posted in Uncategorized | Leave a comment

JBLAS and old FORTRAN

I’ve been doing some work that required some more heavy lifting from a Linear Algebra library than what Apache Commons or JAMA could offer me, and thanks to this extremely handy-dandy, but a bit out-dated website, I found out about JBLAS, a (mostly) JNI wrapper for the BLAS and LAPACK libraries.

I work on Fedora, and the currently supported versions are 26 and 27. The current version of JBLAS (1.2.4), or rather BLAS, has a dependency on libgfortran.so.3, but the libgfortran package included in the Fedora 26 distribution (as of this writing 7.2.1-2) only has libgfortran.so.4. What to do?

The solution is to install both the old and new libgfortran libraries side-by-side.

I downloaded the old rpm from rpmfind.net  and with the command:

sudo rpm -ihv --nodeps --force libgfortran-6.4.1-1.fc25.x86_64.rpm

I was able to get the package installed and JBLAS to run expected.

java -server -jar jblas-1.2.4.jar
— org.jblas INFO jblas version is 1.2.4
Simple benchmark for jblas

Running sanity benchmarks.

checking vector addition… ok
— org.jblas CONFIG BLAS native library not found in path. Copying native library from the archive. Consider installing the library somewhere in the path (for Windows: PATH, for Linux: LD_LIBRARY_PATH).
— org.jblas CONFIG ArchFlavor native library not found in path. Copying native library libjblas_arch_flavor from the archive. Consider installing the library somewhere in the path (for Windows: PATH, for Linux: LD_LIBRARY_PATH).
— org.jblas CONFIG Loading libjblas_arch_flavor.so from /lib/static/Linux/amd64/, copying to libjblas_arch_flavor.so.
— org.jblas CONFIG Loading libjblas.so from /lib/static/Linux/amd64/sse3/, copying to libjblas.so.
checking matrix multiplication… ok
checking existence of dsyev…… ok
[-0.210656, -0.640445, -0.451188; -0.509085, -0.116445, 0.796815; -0.807515, 0.407556, -0.398408; 0.210656, 0.640445, -0.052780]
[17.233688; 1.414214; 0.000000]
[-0.470605, 0.782218, 0.408248; -0.571449, 0.082339, -0.816497; -0.672293, -0.617540, 0.408248]
[17.233688; 1.414214; 0.000000]
checking existence of dgesvd…… ok
Checking complex return values… (z = -21.0 + 88.0i)
Check whether we’re catching XERBLA errors. If you see something like “** On entry to DGEMM parameter number 4 had an illegal value”, it didn’t work!
checking XERBLA… ok
Sanity checks passed.

Each benchmark will take about 5 seconds…

Running benchmark “Java matrix multiplication, double precision”.
n = 10 : 2.596 GFLOPS (6491027 iterations in 5.0 seconds)
n = 100 : 2.923 GFLOPS (7309 iterations in 5.0 seconds)
n = 1000 : 2.536 GFLOPS (7 iterations in 5.5 seconds)

Running benchmark “Java matrix multiplication, single precision”.
n = 10 : 2.482 GFLOPS (6206108 iterations in 5.0 seconds)
n = 100 : 2.408 GFLOPS (6022 iterations in 5.0 seconds)
n = 1000 : 2.381 GFLOPS (7 iterations in 5.9 seconds)

Running benchmark “native matrix multiplication, double precision”.
n = 10 : 1.934 GFLOPS (4835014 iterations in 5.0 seconds)
n = 100 : 8.372 GFLOPS (20931 iterations in 5.0 seconds)
n = 1000 : 12.484 GFLOPS (32 iterations in 5.1 seconds)

Running benchmark “native matrix multiplication, single precision”.
n = 10 : 2.034 GFLOPS (5084278 iterations in 5.0 seconds)
n = 100 : 14.752 GFLOPS (36880 iterations in 5.0 seconds)
n = 1000 : 24.376 GFLOPS (61 iterations in 5.0 seconds)
— org.jblas INFO Deleting /tmp/jblas4508559549457543065/libjblas.so
— org.jblas INFO Deleting /tmp/jblas4508559549457543065/libjblas_arch_flavor.so
— org.jblas INFO Deleting /tmp/jblas4508559549457543065

Posted in Uncategorized | Leave a comment

[Book Review] Leonardo’s Brain by Leonard Shlain

Leonardo's Brain: Understanding da Vinci's Creative GeniusLeonardo’s Brain: Understanding da Vinci’s Creative Genius by Leonard Shlain
My rating: 4 of 5 stars

What a great idea for a book! A mixture of History, Art History, Psychology, and Brain Science, Leonardo’s Brain is a wonderful attempt to try to get inside da Vinci’s head. The book makes a convincing case that da Vinci was the first real scientist in addition to being the greatest artist who had lived up until his time, and details many of the groundbreaking scientific ideas and artistic techniques he originated. Each chapter is an easy read walking through various aspects of what is known about da Vinci with respect to his life at the time, (homosexuality, vegetarianism, etc.) and how that must have been received by his peers and patrons. The chapters also cover various aspects of what is known about brain function and tries to map these onto what is known about da Vinci. Towards the end of the book Shlain starts to reach into a bit of mysticism with claims that da Vinci had ESP in the form of “remote viewing,” which diminishes the quality of the book (hence four instead of five stars). Shlain’s writing style is very readable and touches on many additional topics, which is sure to fire the imagination of the reader and encourage further reading about the unique genius that was Leonardo da Vinci.

View all my reviews

Posted in Book Review | Leave a comment

Microsoft: EmbraceR, ExtendeR, ExtinquisheR

I read with great interest the new blog post over at R Revolutions about the release of “Microsoft R Open 3.3.0.”  And for those of us old enough to remember the Browser Wars of the late 90s, and the resulting  FINDINGS OF FACT concerning Microsoft’s abuse of monopoly power, the first sentence is really eye-catching:

Microsoft R Open 3.3.0, ClarkHeadthe enhanced distribution of open source R, is now available for download for Windows, Mac and Linux systems.

“enhanced.” That sure does sound a lot like “extend.” As in Embrace, extend, and extinquish. Since we’ve already seen this movie, with the only slight plot change, where  there’s a bit of “open” to it, we already know what the intention is. With this admission, Microsoft is either being very honest, or cocky, or just lazy.

I use R Studio when I have to use R. (I love ggplot2!) But in general I hate a lot of things about R. And now when I have to go full-contact script-mode, I have more reason to use Python every day.

Posted in Uncategorized | Leave a comment

[Book Review] The Master Algorithm by Pedro Domingos

The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our WorldThe Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World by Pedro Domingos

My rating: 5 of 5 stars

While coached as a guidebook to help find “The Master Algorithm,” the one AI algorithm “that will rule them all” (his words, not mine), this book is much, much more. At times written whimsically, and at times treating very advance material in a way that non-sophisticated readers can understand, the book is part history lesson, part cultural commentary, and part description of the scientific process. I work exactly in the field of Artificial Intelligence and Machine Learning, and I am definitely a member of the target audience. Maybe I’m just an ML fanboy, but I found the treatment of everything, including that of the author’s own work, to be at just the right level to keep the non-specialist interested, while informing the specialist about those other areas, where he is not a specialist.

The book finishes with musings about what it means to be a source of data for the corporations (and governments) that would use that data for good and/or evil. This alone makes the book worth reading. Such candor from those in the field of machine learning is really refreshing.

View all my reviews

Posted in Uncategorized | Leave a comment

Microsoft and LinkedIn: Probably not good. Maybe not bad.

Oh, what to do about the Microsoft and LinkedIn deal? You may decide to delete your LinkedIn account, like Cathy O’Neil did. Or you may just decide to keep it, although unless you’re looking for a job, or looking for newhires, LinkedIn doesn’t really seem to have much value. I haven’t decided to delete my LinkedIn account–mostly because the data is already there, and even if I delete it, I’m sure it’s just “deleted.” I can choose not to use LinkedIn anymore, but I actually do have friends and contacts on LinkedIn, who aren’t on any other social media platform. Maybe I could start using email again. (Do the kids even call it “email” any more?)

When Microsoft bought Skype, it didn’t make any sense. The technowags at the time were calling it an attempt for Microsoft to appear cool, or monetize subscriptions or… they, like I, were just scraching our heads. Mostly, I thought it was stupid move by a company that had long lost its way. Maybe they had a crystal ball that would point to Messaging being the OS of the future. A little later, we found out that Microsoft, unlike Skype, was reading everything you wrote. Even later we found out that the real reason was that the NSA needed somebody to allow them into encrypted Skype, which at the time was a Swedish company. And what better way than to strongarm an American company rich in cash and that earns a large amount from government contracts? Maybe Microsoft was the only one dumb enough or amoral enough out of the Facebook/Google/Amazon/Apple/Microsoft techopoly to actually do it. It’s not like Microsoft hasn’t demonstrated a lack of business ethics before. Oh, well. That data got sucked into Prism long ago.

If Microsoft really does Clippy-ify LinkedIn, I’ll certainly delete (or “delete”) my account, but Skype is the only Microsoft product I actually use. I got off closed source OS train a long, long, time ago. I don’t really expect to be personally affected by the future integration.

“Predictions are hard, especially about the future.” My suspicions about the last stupid acquisition turned out to be not all that far of the mark. My suspicion about this acquisition is that it’s not nearly as nefarious: Microsoft wants social network graph data to work on and figure out how to monetize in a way that LinkedIn wasn’t able to. No matter what, LinkedIn was an American company subject to National Security Letters, so that data’s been sucked into Prism long ago, too.

Microsoft Research has spent a lot of money to get top researchers hooked on the corporate teat. Maybe the Microsofties just needed a new playground.

 

Posted in Privacy, Social Networks | Leave a comment