converting messy data formats

adde/universal conversion via screen scraping
adde/convert clonespy links
adde/browser/filename cleaner

auto-save with merge versions

3.9: news.adde/camelot's implicit save command:
. camelot uses pyqt, activeMQ, Elixir (interfacing SQLalchemy),
to build apps faster by adding gui info to the model
like django admin interface does .
. they removed the save button 2 years before apple did
and explain this in their faq:
. if you don't like it, take a week off to get used to it .
--
. when the user asks for the save command
it should explain that it is already auto-saving versions,
and that when you hit save(merge versions),
that is deleting the old versions
(undoable until closing session).

1st-class subfile renaming

12.4: news.adde/fs/1st-class subfile renaming:

. Brandon Casey June 20th, 2007,
mentioned the idea of 1st-class subfile renamingwhich is a version tracking system that recognizes
when file modifications consist of subfile moves:
"( I split the file into two files in my repo.
File is modified in source repo.
I retrieve changes from source repo.
Will changes be correctly applied to
relevant sections in the two new files?
).
. Casey was responding to an article about
www.markshuttleworth.com/archives/123
tracking renames as a first class operation,
which I would call 1st-class file renaming;
so, Casey's idea is 1st-class subfile renaming .
. the minimalist style of version tracking
is to wait for a commit,
and then respond to modifications:
# directory changes are seen as
file {adds, deletes};
# file modifications are recorded with a
line-oriented diff' file .
. in previous notes on subfiles,
the system recognizes structure within files:
eg, html files have header tags, H1, H2, ... H6;
and, each represents a hierarchical subfile,
as do paragraphs, divisions, and other enclosing tags .
. the file system is just a tree of pointers;
likewise, this tree extends to subfiles;
and files become pointers to sections of text
which are actually in a database
named by their creation date & author
(plus an optional serial# for when
several subfiles have the same create date;
for example when a file is modified by
having some paragraphs inserted,
the paragraphs' create dates are considered to be
at the file's modified time .
).
. the branch copies db names
and each simultanious branch belings to one author;
so, that author'name must be part of a subfile's db name .

1st-class subfile renaming:

. the folder-subfile system is a tree:
a collection of nodes
(visualized as forks for representing enclosures )
and edges
(visualized as branches for representing
the things they contained by an enclosure ).
. a folder-representing node is thus remembering
# the name of the folder,
# edges to all nodes the folder contains;
likewise, a file-representing node has the file's name,
and the edges to subfiles it contains;
but, a paragraph-representing node
has a position number in place of a name
( so, when a paragraph gets inserted,
that bumps all the position numbers of paragraphs below
(within that subfile not the entire file)).

. when version-tracking for subfiles,
the nodes that were revised
are replaced with version-representing nodes
(version node) with edges to
both the replaced node,
and to the new node created by the revision;
in summary,
modified nodes are presented as a menu
of version choices;
eg, a folder-move creates version nodes twice:
once at both the source and destination nodes .
12.5:
. what if the version node is not the smartest
way to see changes?
the friendliest way is to show the various views:
# toc's for each version:
when there a high use of 2 views,
make sure both are fully built (not defined as a script)
the db will then be like a book with 2 toc's (table of contents).
# index:
. search of index finds every instance
showing you the pathnames,
and grouping together occurrences of
1 object is in several locations across versions .
# glossary:
. the place for histories of how a name has been used,
and why it has been changed from and to other names .
. this would include the human view of
why there systematic renames in certain versions .

optimizing timemachine

12.4: adde/fs/optimizing timemachine:
. mac's timemachine can prevent corruptions;
adde's version of that could additionally preserve memory
(and thereby store more changes)
by analyzing how things change,
and asking the user why something got deleted .
--
. the inspiration for this idea was that
my data had be corrupted by a fat32 sd card error;
and although timemachine did save me,
all I really needed was
some automatic analyst to notice strange behavior:
the user fills up log entries,
but here one has been deleted ?
a folder had many files but is now null ?
12.5:
. the prior, simpler idea that helps as much
is to keep all user contributions in a read-only db,
and saving memory by automatic duplicate removal
(a file can saved as a script such as:
include these 3 subfiles,
and then replace phrase x with y ).
. adde questions the user upon each deletion:
why is this deleted?
# somewhere else: deleted when verified;
# not yours : did you want to log this? source?
# duplication:
adde has automatic duplicate removal .

tags, link-trees, and file-trees

7.3: 7.30: adde/tags, link-trees, and file-trees:
. how to integrate tags with folder systems?
placing a file in a folder is tagging it;
 a hierarchy is a tag with segments;
ie, each path name is one tag .
. while a file can be in only one folder,
it can be assigned to many tags .

. tags can be thought of as
a list of links to its members;
and, equivalently, as attributes of a file:
each file or file link can list all the tags
to which it is a member,
just as it can tell the folder path it's in .

link-trees vs file-trees:
. a file-tree is the usual folder tree,
it contains actual files rather than links;
so, a file can be a member of only one file-tree .
. the link-tree is like a file-tree
except it contains only links to files
rather than actual files;
thus, a file can belong to many link-trees,
and may be in many places
within the same link-tree .
7.30:
. the main idea for these distinctions
is that file-trees are low-level concerns;
the only reason they need to be accessed
is when the space limits require
deletions or archiving .
. most file system operations
are concerned only with organizing files,
not managing limited space,
and each activity should have its own
customized view of the file system;
so, most file accesses should be done
through link-trees .
. tags can be applied to either files or links,
depending on the tag's scope:
tags on files are global;
tags on links apply only to
a specific link-tree;
ie, each link-tree has its own set of tags:
both the ones induced by the global tags
belonging to files it's linking to,
and the tags specified by the
user of the link-tree .

compare databases, filesystems

1.24: adda/oop/db:

5.4(2009): adda/oop/db:

. how is a fs different than a db,
eg, how to impl' db with fs?
. the db controls how it stores its data as files;
but one way is
a record gets a folder named with a serial #,
then each file is named after a field in the record;
if that field is structured
then it would be a folder named like a file with a d'type name
. another subfolder system holds oop'ish meta data .
. so, instead of using db mgt to access records,
it sends msg's to a record`s type mgt
so data formats can change without reprogramming db mgt
or normalizing db .

. the oop'ish question to ask is
what operations exist for db vs fs ?
. the unit of a fs is the file
which is a db restricted to a few components:
(name, owner, type, creation.date, mod.date, etc)
. the unit of a db is a record
whose components are specific to attributes of
what the record is a symbol of .

filesystem as intranet of vol's and slots

10.13: mis.adda/syntax/directory as pointer model:

. if the ptr syntax is modeled after files,

then you bump into the confusion that

although folders are impl'd as pointers,

they have {copy, cut, paste} operations

that are acting as if they were containers

just like arrays, or records .

10.17: adda/syntax/volumes as a domain class:

. to unify the files systems of the pc and unix,

removables (type"volume) should be considered to be

internet nodes of domain"vol .

. a user has, both a current vol

and a current directory within each vol .

[typical pc behavior?]

10.26: adde/fs/entity and its id:

. another dimension of file id

is the volume the file is on;

because, an owner can write to several volumes

at the same time

so is not unique .

in fact, the vol'mgt is then

both the unique and direct creator;

but, consider every file must be sync'd across multiple val's

so, like git,

the unit of creation should be considered to be

the project or db, not the file .

10.27:

. a file can be uniquely identified by

(vol name * create`time * serial#)

. the serial# is needed if the vol can create a

set of files at one time,

or if the system clock is not giving accuracy to put

every file-create in its own time slice .

content vs container:

. the create time for the file

differs from that of the contents .

10.27: adda/syntax/url` slots vs vol's:

. the url domains: {vol, slot, drive} are for local content

that is reachable by a removable disk .

. drives are plugged into slots,

and volumes (eg, disks) are plugged into drives .

11.5: adda/syntax/url/filename recursively like site name:

. suppose net syntax is recursive,

so a file can be another whole net system:

//x.com/y.com

. the first idea to flesh out is

how it could make sense to use dot notation for files

the way sites do:

subj.name.type

. when they had www.x.type,

the www (front-dotted name) meant an entity

had a library fit for world-wide webbing

. another view is that

www refers to the default app;

eg, www.google.com means google's main service (search);

but then there's other services

mail, picasa, knol, code, etc .

. sites are {processors, entities} offering various services,

so in the case of locally,

a site-style name could be used for users:

eg, dev.addn.user would be a file-system dedicated to

addn's dev'ing .

11.12: adda/syntax/url/filenames like sitenames:

. when filenames have types like websites domains,

that means they are processors,

and their domains have similar meanings:

.com is closedware,

.org is openware,

.edu is knowledgebase or expert system,

.mil is security services,

.gov is os daemons .

12.1: adda/syntax/url/the register domain:

. an url naming system should represent all address spaces as

internet nodes;

eg, registers are separate address space

and so they should have their own website in the reg domain .

. the url's protocol is really not needed;

it is indicating the communications type

and is esp'ly irrelevant when the os is the same everywhere;

but, even with a variety of os's

it seems the usual is https, and that should be the default .

12.1: adda/syntax/url/the internet root as module symbol:

. a corollary of every address space getting a web node address,

is that the internet root symbol (//)

is the prefix that identifies concurrent objects:

eg, task instances, and sharable (stateless) library units .

. additionally, (//) identifies a pointer to filesystem

such that you can have access to multiple filesytems,

each having a current working directory;

eg, //destination`= //source .

whereas, //source/ (with the (/) appended)

would indicate the root directory rather than the current working directory .

. the domain name is a type name,

so then name.domain (where the type.mark happens to be a domain name)

would declare name to be a filesystem .

. once you know that a symbol is a domain-typed obj,

when is the syntax (//name) needed ?

for recursive use of domain names,

it shows when the symbol at top level

-- the level of the internet or the system

(some of the domain names are reserved for system components) .

type a/.//?

. the // is part of address literal

so not a type name as its ambiguous

. just as (/) means ptr concept

(//) symbolizes the a modular (or removable) filesystem

a separate address space,

. every tree var is //a.tree/...

adda lib consists of nodes in the adda domain //a.adda/

substructures and aliases

7.28: adde/alias handling:

. an alias is dangling when its target is deleted;

so, any delete operation should involve asking

whether aliases should be deleted also;

or, whether their appearance should be changed to reflect the dangling state:

eg, arrow symbols are changed to null, and name greyed .

. copying across volumes will copy the link .

. adde should also make a backup of link`targets;

and, also address complications of

duplications, and sharing updates, etc,

ie, when copying across volumes .

8.3: links are of 2 types:

. one is invisible so that any copy to the unit

is going to copy what is pointed at by the invisible link .

. the visible link is an url that means it is external to the unit;

and so should be copied only if wanting to copy multiple units .

. hence there needs to be some way to describe what a unit is;

like git identifies a project as the unit .

7.28: adde/subtructuring with modular skeleton:

. an efficient way to modularize a file

is to put the structure of it

(the table of contents with links to pararaphs)

in one file (the skeleton subfile)

so that changing a subtitle opens just the skeleton subfile .

unique filenaming

6.21:

. in a virtual fs (file system),

users can have any filename, virtually visible in multiple folders,

but the actual files are all named after

both the creation date, and the user name .

. the user is identified by the owner of

the super folder that all the files are in .

. if there is more than one user working on the fs,

the file names are unique because they reside in a user folder .

[7.7:

I got this idea from wondering how to share the addx project:

the project can have each author have their own version

in a separate subfolder .

. parts they decide to share are outside of the version.folders .

]

7.7:

. when working in a multitasking and multiprocessing system

with multiple agents,

the name is only unique when using both a timestamp

and the chain of command

(eg, a human starts an agent that employs another agent ...)

so then the name is human`agent#1`agent#2-timestamp

. the id#'s can stay small because a human user has

only a handful of employed agents,

who in turn have only a few employees

(service providers or libraries) .