safer C coding

9: news.cyb/dev.c/avoiding remote code execution:
leafsr:

. some C library functions are often used incorrectly,
and that consequently result in
remote code execution vulnerabilities:
strcpy, sprintf, memcpy, CopyMemory,
RtlCopyMemory, strcat, alloca
and many more Win32 specific stuff.
. much of the legacy software that is
still critical to many enterprises
contains code that calls these vulnerable library functions.
Despite modern memory protections
like ASLR and DEP
the vulnerabilities these functions introduce
are still exploitable under the right conditions .

 . here's the safe C library
– that implements strcpy_s and friends
as an open source library (MIT license).

19: news.cyb/dev.c/SAFECode updates secure dev guide:
.  pubs from safecode.org .
scmagazine:

The Software Assurance Forum for
Excellence in Code (SAFECode),
a nonprofit seeking to advance software assurance,
released an updated guidance document .
The free report includes verification methods and tools
that can be used to confirm whether
development teams have followed prescribed practices.

multi-language to univeral data format (UDF)

addx/multi-lang like .net:
10.2: 12.11:
. the addx system needs to be like
Microsoft's ".NET" such that it lets us
command addx with a variety of languages
-- adda would just be the default language .
. a new language being ported to addx
would provide a compiler that emits
addm's high-level code (HLC)
that is similar to .NET's assemblies .

extension modules

19: 21: adda/extension modules:
. Python' C interface has 2 uses:
one face simply models python
(for calling python from c),
whereas the other face is for writing extensions:
calling c from python .
. the primary purpose of adda,
on the other hand, is to avoid C:
all extension modules are written in adda,
and then adda translates that to C,
like what Cython is doing: Python to C .

PLY(Python-based Lex-Yacc) and pyCparser

8.20: web.adda/dev.py/ply
11.2: PLY compared with ANTLR:

There are several reasons to use ANTLR
over one of the Python parsers
like PLY and PyParsing.
The GUI interface is very nice,
it has a sophisticated understanding of
how to define a grammar,
it's top-down approach means the
generated code is easier to understand,
tree-grammers simplify some otherwise
tedious parts of writing parsers,
and it supports multiple output languages:
its builds parsers best in C, C# and Java;
but also has some support for generating Python .

ANTLR is written in Java;

Unlike the standard yacc/lex combination,
it combines both lexing and parsing rules
into the same specification document,
and adds tree transformation rules
for manipulating the AST.

. so, if you prefer python tools over Java, PLY ahead!

ways of referring to non-locals

7.28: adda/function/ways of referring to non-locals:
[8.8: intro:
. when namespaces are nested
such as when a program contains subprograms,
the usual situation is called shadowing
which means, for example, that if
subsub is within sub is within main,
and they each declare a symbol, x;
then each scope sees only its own x .
. if subsub wanted to see main's x,
it might specify a pathname;
eg, global/main/x .
. we might also require this for all globals
just to make it easy on the reader;
because if symbol y were not defined in subsub
yet subsub was using it,
then  we have to look in every superscope,
(sub, and main, in this example).
whereas, if we require pathnames for globals,
then the reader instantly knows the meaning of
global/main/sub/y .]

. instead of refering to a scope's ancestors
to reach globals,
we could use "(global) treated as root of fs
to reach non-locals .
. "(parent) is useful as a namespace root,
but if you're going to have a parent.keyword,
users will expect a grandparent as well
(gparent, ggparent, gggparent, etc) [8.8:
so that instead of writing global/main/sub/y
it can be compacted to gparent/y .]
...
. of ways to identify globals,
it was suggested that we could recognize
such roots as {global, parent, gparent, etc};
but in our case,
the global namespace will be static
(no symbols are added after compile time);
so that gives us an advantage
that will make it easier on the writers:
all they need to do is find
some unique subset of the url,
and then we can identify the full url
by doing a bottom-up search
comparing the leaf first . [8.8:
... however, chances are,
that is hardly reader-friendly;
if the name seems long, and it's used much,
then use renaming: ( x: gparent/x; ); 8.9:
nevertheless, we can have the compiler
use this system for helping both readers and writers,
by letting writers abbreviate, and then
expanding abbreviations to help readers .]

generic parser

2.23: adda/translate/generic parser:

. if there is a generic parser,
then it has to record spaces(n) and new lines,
because the meaning of symbol sequences
will often depend on whether a space
is separating them .
. this may not help me anyway since my interpretation
can depend on whether a name is a type id or not?

. one thing that does simplify things
is a parser that does find and tree
the enclosures and delimiters
-- {} () [] , ; : .  --
along with some other un-redefinables .

. it also supports implicit enclosures
(the use of indentation to indicate a grouping)
so it needs to translate ( newline & (some spaces) )
into the beginning or continuation of an enclosure;
and, there needs to be a parameter for
what the currently expected indenting level is .

. if it can find (symbol.type) as being a type def,
then it can also find ( .symbol ) as being a type; [2.29:
but, types shouldn't require a preceding dot .]

. strings of alphanum's that start with alphabet
are always names .
. enclosures include set generators .

parser uses string descriptors

3.3: adda/translate/parser/use string descriptors:
. one way to parse a string
is use the lexical analyzer to find the relevant substrings,
and then make lexical nodes that point at copies of string;
the way I'm thinking of now
uses string descriptors instead of copies:
put the string in an array of characters,
and then a descriptor is an index into that array,
along with intended length .
. the advantage to this way is that
if the parser gets confused,
a repair algorithm can work with the result
searching not only the tree,
but also the string,
and seeing how they relate .

passing destination references implicitely

12.9: adda/translate/passing destination references implicitely:
. by passing destination locations as references
instead of having functions return objects,
we can avoid a lot of garbage generation,
but the reason oop doesn't do this is it complicates the job of
application programmers .
. the reason addx does do this, is it has a translator
that does this automatically for programmers:
the compiler is looking for their y`= f(x)
and converting it for them into f-proc(&y, x).
. I think I had a contradictory example somewhere,
but it only applies to when the library programmer is
wanting to do systems programming (when it extends the compiler).

including obj'c features

7.23: adda/oop/including obj'c features:

. in the expression"( a`b(x)`c ),
the ( a`b(x) ) means
the object"a has a procedure b(x)
that returns an object location
whose available functions include c .
. the syntax"(a`b) is not necessarily
involving self-modification;
it simply means subprogram"b has access to (a)
as an inout parameter
(the subprogram's object parameter).

. is there some efficient way of doing a`b`c
without passing pointers ?
I was thinking there was some virtual way
that doesn't really need pointers
(oop doesn't really need
all the pointers it thinks it needs)...

. if a subprogram's object param
returns a reference to something other than
the object param,
then in the c translation of same
this subprogram takes a destination param;
ie, a param that inputs a destination address
showing where the caller wants the return placed .

. why wouldn't it always take a destination param
even when returning self? ...
because the caller is already housing that obj .

7.26:
. my original concern here
was allowing the passing of references
and the use of object param's,
just like obj'c oop does,
while still using the sort of oop system
that did not generate garbage .
todo:
. I think obj'c has the convention that
even if ( a`b(x) doesn't return anything
if an object reference is expected as a return,
as is the case in "( a`b(x)`c ),
then it will use a reference to (a);
ie, ( a`b(x)`c ) would equal:
( a`b(x); a`c ) .

avoiding freezes with real concurrency

5.21: 6.2: adda/translate/co
there were 2 cases of multi-tasking:
# gui reactor:
. the multi-tasking done for gui response
could really be done by
frequent subroutine calls to the gui handler
instead of swapping threads with it;
because, its endless event loop is equivent to
an endless number of brief loop body calls .
# time-slicing:
. when a program is composed of threads (co.programs),
it expects them to be time-sliced,
so that, for instance,
if there are several things to animate,
each refresh of the screen is showing
what appears to be concurrent activity .
. unlike the gui-reactor case,
there is no way to use a subroutine call for
sharing cpu time with anonymous threads;
because, generally they cannot be decomposed into
an endless number of brief loop body calls .
. a similar case is anonymous co.programming,
where the user runs several programs at once;
this situation is significantly unlike
app-orchestrated co.programming because
user-launched co.programs aren't aware of each other;
so, even if c did support coroutines,
they would not be of use in time-slicing here .
5.22:
. another place where programs can hang
besides infinite looping and recursion,
is waiting for devices to respond;
the addx library designer needs to know
where these hangups are,
and not give tasks direct access to them .
. instead,
the sub is suspended until the wait is over;
and control is given instead to gui reactions .
[6.2:
. it may be the case that
only hardware-based concurrency could provide
the needed protection from device freezes;
that means the base language can't be just c,
but involve c+posix, obj-c + gcd, or qt .]

5.29: web.adda/translate/co/open concurrency api's:
. unix is a std open platform now,
isn't there some concurrency api defined in posix?
. an overview of user-level vs kernel-level
and threads vs processes:
. processes communicate via sockets
pos:
. addx's primary use for threads is that
the main thread is adde dishing up the gui,
and it sends messages to a side thread
for housing the apps that do the actual work .
. while getting bogged down in posix idioms
it occured to me that since my first target is mac,
I should follow what they suggest
in the way of concurrency primitives .
...
there's also qt:
both open, cross-platform and easier than posix .

programming in-the-large with c

8.5: news.adda/translate/info'hiding n type-safe linking:

2007 CMod Modular Information Hiding
and Type Safe Linking for C

CMod, provides a sound module system for C;
it works by enforcing a set of rules that are
based on principles of modular reasoning
and on current programming practice.

CMod's rules safely enforce the convention
that .h header files are module interfaces
and .c source files are module implementations.
Although this convention is well known,
developing CMod's rules revealed many subtleties
to applying the basic pattern correctly.
CMod's rules have been proven formally
to enforce both information hiding
and type-safe linking.

rapid compile times

11.10: adda/translation/rapid compile times:
. during r&d people want quick compiles
(that's what inspired google to create Go.lang);
whereas, one problem with adda
is that in making the lang
easy for the user, and feature-rich,
its compiler might take a long time
trying to figure out what the user wants .
[11.12:
. one remedy for this is integration with adde,
which calls the compiler to attempt tentative translations,
and compiling in sections,
so that if a routine compiles,
and just one line is modified,
only that one line needs to be recompiled .]
. adda can also speed compiles by skipping c,
and translating to addm (the virtual machine).
. addm can do all the safety checks as needed,
rather than at compile-time .

gooey gui portability

6.28: pos.adda/dev.gui/stay with the mainstream tools:

. sdl (simple directmedia layer)

reminds me that being low level
could get you into trouble;
the original adda plan
to simply write a native code compiler
for each system, might be
not only cleaner, but also safer,
since a platform's own code and tools
are likely to take better care of you
if not with a safer language,
then at least with documentation
that warns you where the landmines are .

. a browse through sdl-style portability
points out that c is not multi-core .
. the goal of portability
-- as in maximal c coding --
could interfere with some
very smart c concurrency extensions .
. if concurrency should be pervasive,
your code will be riddled with non-std c
(don't even bother trying for portable).

[6.30: but the mvc pattern could help with

turning concurrency into a module? ]

plain old c ... minus

6.25: pos.adda/plain old c ... minus:
[6.30:
. google's native-code sandboxes
in Android and in Chromium OS
are an important platform,
and should be considered before
deciding what "std" C is .] [6.28:
. many platforms will restrict the std libraries
in order to further enhance security
or adapt to smaller, energy-wise situations .]

. keep in mind what's available in
the android ndk:
libc (the C library), -- as Bionic, not glibc.
JNI interface, -- controller-view of your mvc;
OpenGL ES {1.1, 2.0} (3D graphics)
libm (Math),
libz (Zlib compression)
liblog (Android logging)
libjnigraphics (JNI Pixel buffer access)

. the API is included with the sdk;
so, get that installed on ubuntu

(eclipse runs better there than mac?) .

details of android's custom libc:

. customizing libc (as Bionic)
was necessary for three reasons:
* License:
keep GPL out of user-space.
Bionic code uses the BSD license.
* Size:
Bionic is half the size of glibc
* Speed:
custom pthread implementation.
Bionic has built-in support for
important Android-specific services
such as system properties and logging.

It doesn't support certain POSIX features,
like wide chars (wchar_t)
-- The world has moved on to Unicode
with its various fixed and variable
width encodings,
which the wide character type
is not particularly useful for.

. use ICU for internationalization support,
(International Components for Unicode)
instead of LOCALE .
ICU is a mature, widely used set of
C/C++ and Java libraries providing
Unicode and Globalization support
for software applications.
ICU is widely portable
and gives applications the same results on all platforms
and between C/C++ and Java software.
threads
The bionic C library has it's own thread API,
not the same as either original LinuxThreads or NPTL.
It implements only support for Dalvik JVM,

Mutexes, rwlocks, condvars, etc
are all implemented using kernel futexes,
-- see Ulrich Drepper's futex whitepaper.
There is no pthread_cancel().
Threads can exit,
but can not be killed by another thread.
There is no pthread_atfork().
Thread local storage is implemented,
with up to 64 keys handled.
Android reserves several of these for its own use:
the per-thread id and errno,
as well as two variables related to OpenGL
whose function I do not understand.

stdio, stdlib, string, unistd:
. removed the LOCALE support from strtod()
(i.e., is the decimal point a period or a comma?
In the Bionic library it is always a period).

{openlog(), syslog()} are replaced with
__libc_android_log_print()

Bionic uses Doug Lea's malloc, dlmalloc.
Bionic also provides a hash table
to track allocations
looking for leaks, in malloc_leak.c.

no asynchronous AIO routines
aio_read() or aio_write().

no crypt()
-- uses OpenSSL for that .
. dispenses with most file-based Unix administration.
no getfsent,
because no /etc/fstab.
. there is a /var/run/utmp,
and so there is a getutent()
Android implements its own account management,
and does not use /etc/passwd.

. no getpwent(),
and getpwnam()/getpwuid() are implemented as
wrappers around an Android ID service.
At present, the Android ID service consists of
25 hard-coded accounts in android_filesystem_config.h

. no termios support (good riddance).

Bionic is a BSD-based libc with support for
Linux system calls and interfaces.
If limitations prove untenable,
the syscall and pthread mutex implementation
could be repurposed into the heavier
FreeBSD/NetBSD/OpenBSD libc,
though handling thread cancellation
while using the Bionic mutexes
could require additional work.

oop theories, designs, implementation notes

6.1: adda/oop/based on vari-sized records:

. the 3 sorts of inheritance:
# subtyping:
. inheritors are assignment compatible .
# symbolic description:
. describes one type in terms of
another (incompatible) type .
# generics:
. an object's function is selected by
the function's name (or hash of that)
-- rather than referring to methods by
index, pointer, or offset .
-- this is the obj'c meaning of inheritance; [6.7:
obj'c also supports subtyping . 6.30:
... actually,
any system that supports generics,
implicitely supports subtyping .]
. after review oop ideas,
I wondered if the the trailer idea
would be useful for oop subclassing .

. parameters that vary in size
are accommodated by replacing value with
a mini-pointer (an index into
an associated trailer array)
that then leads to the value .
. a vari-sized record is organized as
a head (the mini-pointers)
and body (the trailer).
. in the vari-length record,
the number of fields is fixed,
whereas in multi-inheriting oop,
not only does the body grow
but the head too is vari-sized .
. provide a mini-filesystem (fs)...
[6.30:
. each class expects to find
it's own instance variables
arranged in the usual way
even when it's one of many ancestors
of the obj's class .
. in a mini-fs, the directory
doesn't have to be on the obj;
just containing a type.tag,
will be eno', since that is a
key to using the lib mgr's map:
(instance's class, ancestor.class) ->
offset into instance .
-- this map shows what offset to use
into the instance
to get to an inherited class's
embedded instance record .]

6.30: todo.adda/oop/obj'c dynamic linking:
. when there is multi-inheritance,
you have an instance variable where
there are separate sections
for each subclass's privates .
. it would be efficient if type-mgr is
passed ptr's to the operated obj's;
and generally,
how is each type-mgr supervised
so as to insure that they
don't affect any surrounding bits?
. you trust your compiler,
but then in obj'c,
you can relink modules dynamically;
and, the new module might be
code from another compiler .
. need to review obj'c
for how to best use,
and not be used by, that subsystem .

6.5: adda/oop/how does c do obj'c dynamic bindings:
. the generic-oop functionality of obj'c
comes from being able to launch a subrogram
when given a function`name and a type.tag;
this could be done in unix by calling to the os
to execute a shell command;
. without that ability,
each addition to the class library
would require editing the obj'c run-time engine
and re-linking it with the expanded class library .

6.7: adda/oop/subclasses as subtypes:
. what is a new type, vs a subtype, in adda?
subtypes can't remove or ignore
inherited functions,
but new types can do a reuse but remove;
how does Ada do this ?
inheritors are new types (meaning it's
not assignment-compatible
with the inherited type);
if you want a variable to have subtype polymorphism
(ie, be assignment-compatible with
all inheritors of Baseclass),
then you need to declare the var's type to be
Baseclass'class .

6.23: pos.adda/obj'c and cocoa bindings:
. the reason to not interface cocoa
is that it's very inefficient with its
ref'counting and its (typical) way
of implementing oop .
. to the extent there does need to be
a binding to it (for the sake of
getting the usual services)
the binding will not be direct:
the mac gives all the power to the developer,
whereas addx will give the user
most of the gui design .
. for example, in nu,
"( locally-scoped assignments are
instances of NSMutableDictionary; )
this might look like an
excellent chance to reuse,
but you might be using thousands of those,
which means juggling a lot of within
both your subsystem's heap
and the mac's system heap .
. it might be ok for prototyping,
but eventually, you want basic services
to be based on your own mem'mgt .

nui (natural user interface)

4.14: news.adde/nui (natural ui):

Don’t Over-Interpret Apple: Cross-Platform Development Isn’t a Sin
(a reference to Apple's disallowing code generators)

. Intua BeatMaker looks native,
but this app is built with a
cross-platform library, nui:
the multiplatform C++ API
for 3D hardware accelerated GUI .
. Yapuka.app is a NUI project generator for Xcode .

NUI (natural user interface):

. the GUI relied on metaphors for
interacting with on-screen content or objects.
The 'desktop' and 'drag' for example, being
metaphors for a visual interface
that ultimately was translated back into
the strict codified language of the computer.
The natural user interface removes the metaphors,
and many of the artificially learned devices,
to allow users to more directly manipulate content
using more Natural movements, motions and gestures.
. 'intuitively' describes how users interact with it.

. the seminal paper for post-WIMP interfaces
is Non Command User Interfaces .
. Michel Beaudouin-Lafon subsequently proposed
a framework called instrumental interaction(pdf),
that defines a design space for
Post-WIMP interaction techniques
and a set of properties for comparing them.
Examples of Post-WIMP interaction include
tui (touch ui):
. "(tui) can also stand text (lexically-graphical) ui;
it should be renamed to avoid confusion with touch ui; [5.1:
lui (lexical ui)? ]

cross-platform means a deep binding

4.11: news.adda/cross-platform means a deep binding:

summary:
. Apple is disallowing the use of code
that uses an intermediary translation
or compatibility layer .
. the important thing is that when intelligently done,
intermediary translations are no different than
people translating pseudocode to the target language .
. you'll know when it's intelligent
because they won't be able to enforce it .

rationale:
. this rule is just being fair to
dev's that spend the extra time
to get their code designed for efficiency;
Apple is acknowledging how important it is
to be first to market with an idea,
and if people are creating virtual machines
this will make it easy to be first to market
but at the expense of keeping the app' efficient .
. the xcode system will generate code that follows certain rules,
and it would not be easy to quality-check the app'
unless the code is following those rules .

rationale @ daringfireball.net

. what Apple does not want
is for some other company to establish a
de facto standard software platform
on top of Cocoa Touch.
Consider a world where some other company’s
cross-platform toolkit proved wildly popular.
Then Apple releases major new features to iPhone OS,
and that other company’s toolkit is slow to adopt them.
At that point, it’s the other company
that controls when third-party apps
can make use of these features.

co.android alerts dev's to Apple's major change .

daringfireball.net 8 April 2010

. section 3.3.1 of the iPhone Developer Program License Agreement:
Applications may only use Documented APIs
in the manner prescribed by Apple
and must not use or call any private APIs.
. the 4.0 SDK beta now adds:
Applications must be originally written in
Objective-C, C, C++, or JavaScript (iPhone OS WebKit engine)
and only code written in C, C++, and Objective-C
may compile and directly link against the Documented APIs
(e.g., Applications that link to Documented APIs
through an intermediary translation
or compatibility layer or tool
are prohibited).

Steve Jobs explains:

“intermediate layers between platform and dev'
ultimately produces sub-standard app's
and hinders the progress of the platform.”

a response:

. it doesn’t make sense to limit
source-to-source conversion tools like Unity3D

--

unity3d creates a complete XCode project that just works!
. using iPhone Remote,
you can use an actual iPhone or iPod Touch device
to view and test your game live,
right from the Unity Editor. [using javascript on the web]

--
Unity3d's CEO:

. In the ancient days of the App Store (July 2008),
Apple very late changed the kernel to disallow
JIT (just-in-time) compilation.
What we did instead was spend several months
changing Mono to compile scripts
AOT (ahead of time) instead
(this is why some dynamic constructs in our JavaScript
doesn’t work on the iPhone).

my response:
. if Unity3D is disallowed,
at least they are ahead with a source-to-source tool;
now all they have to do is make it a deep binding:
convert their graphics library functions calls
directly into cocoa graphics calls .

4.29:
. the mac#iphone expects code to follow the idioms,
and this is achieved by very intelligent transforms:
adda code -> mac idioms .
. each platform needs its own adda backend;
adda's oop system should not be concerned with
having a shallow binding to various other oop systems .
. adda's front end should turn human ideas into
expression trees that can be analyzed by computer,
rather than using the idioms of a given platform;
and then,
adda's backend for the mac#iphone
should be able to translate human ideas to
iphone`particulars
-- the same way a programmer does
(this is the sort of work lisp was invented for).

4.30: my response after reviewing the issue:

Ludd·ite:
Pronunciation: \App'll\
Function: noun
Etymology: perhaps from Ned Ludd, 18th century Leicestershire workman
who destroyed a knitting frame
Date: 1811
. one of a group of early 19th century English workmen
destroying laborsaving machinery as a protest;
broadly : one who is opposed to especially technological change .
— Luddite adjective .

key to portability is sourcecode binding

3.30: adda/key to portability is sourcecode binding:
. dalvik-vs-mono:

"( J2ME was always crippled by the phone vendors
who never wanted to get important APIs into the stack
so that they can bind the developers to
their implementation.)

. what they are calling an implementation
is simply all that boring translation work;
the first thing you have to do on a new platform
is make a backend for your translator
that will map your preferred language
to their version of library calls .
. this is why having a source-to-source translator
is the most efficient way to portability .
3.31:
. it requires a lisp-like infrastructure
that makes it easy to automate all the rules
that one usually applies to programming;
ie, translating from pseudocode to the target code .

portable translation the pypy way

3.26: bk.adda/Self-Sustaining Systems:
Self-Sustaining Systems: First Workshop, S3 2008 By Robert Hirschfeld, Kim Rose

. pypy offers a model-driven approach
to language implementation:
unlike jython, which diverges from cpython,
pypy is providing a pil (pythonic intermediate lang')
which can then have a backend to every other lang;
ie, the compiler is translating as if
that lang were a cpu intstruction set .
. pypy doesn't try be the intermed'lang for all back ends .
. it has versions for
high level (oop'ish) langs -- eg, {cli, jvm, js} --
and low level langs: eg, {llvm, c} .
. by coding your translator in pypy (Rpython)
you can debug it on the batteries-included cPython system
and then have all these back ends it will run on
-- including c .

. great, now if I could only love python!
3.31:
. this presentation assumes jython diverges
simply because it's rendered in a
different code base;
but jython differs for a deliberate reason:
it didn't want to just be another python
that happened to be runnable on a jvm,
it wanted a lang customized to complement the jvm;
whereas, cpython wants a lang that is
basically customized to complement ms`windows .

key to concurrency support

3.14: adda/translation/concurrency
. the recently found scripting lang, falcon,
has good support for concurrency
-- very impressive .
. might be good code to reuse for adda;
but c++ code and with multiple authors
might be quite frustrating for a
beginner of the lang .
. it doesn't take a lot of coding
with an elegant design;
the key to concurrency
is the recent idea for coroutine translation:
every sub is converted to a coroutine
and a yield placed in every recurrence
(loops, upward goto's ...).