[tug-summer-of-code] Project idea: LaTeX3 "microkernel"

[Joseph Wright]

Hello all,

I've proposed in various places a LaTeX3 "microkernel", and I wonder if
this might be an interesting project for the Summer of Code. The plan
would be to write something based on the existing expl3 code which could
typeset a minimal document:

\documentclass{minimal} % Or something similar
\begin{document}
\emph{Hello World!}
\end{document}

without using LaTeX2e at all. A lot of the work would involve finding
how to re-implement the LaTeX kernel in the expl3 syntax.

At the initial stage, the plan would not be to make major alterations to
the way LaTeX works (a lot of work needs to be done by the team on this
yet). Of course, there are lots of ideas that could be brought to the
project if initial progress was fast (for example, Will Robertson's
suggests about the catcodes of special characters in the document body).

While this would be somewhat separate to the official LaTeX3 work, I'd
hope that the team would be supportive of the idea. It would show up
problems with the current expl3 code, and would hopefully provide some
useful code for the team to eventually use.

Thoughts?

[Will Robertson]

Post by Joseph Wright
I've proposed in various places a LaTeX3 "microkernel", and I wonder if
this might be an interesting project for the Summer of Code.

[...]

Post by Joseph Wright
While this would be somewhat separate to the official LaTeX3 work, I'd
hope that the team would be supportive of the idea.

I certainly support this idea; LaTeX3 is open source, after all, and I
feel that improvements to the code or new ideas generated from such a
project would be welcomed and helpful in the near future of the LaTeX3
project.

I'm a little under the weather to be able to devote much time to this
project, however; I'd be happy to help as best I could, however. Is
there any chance, Joseph, that you'd have the time to mentor this
project if a suitable student signed up? (And if this idea goes any
further we should raise it together with the LaTeX3 project members...)

Will
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2415 bytes
Desc: not available
Url : http://tug.org/pipermail/summer-of-code/attachments/20090308/a3068939/attachment.bin

[Joseph Wright]

Post by Will Robertson
I certainly support this idea; LaTeX3 is open source, after all, and I
feel that improvements to the code or new ideas generated from such a
project would be welcomed and helpful in the near future of the LaTeX3
project.

Thanks. My feeling is that it would make a nice complement to the
low-level stuff that the team are doing.

[snip]

Post by Will Robertson
Is
there any chance, Joseph, that you'd have the time to mentor this
project if a suitable student signed up? (And if this idea goes any
further we should raise it together with the LaTeX3 project members...)

To be explicit, I hoped I was implying that I would mentor this, if
taken on, as it is my idea :-)

[Karl Berry]

Hi Joseph,

I've proposed in various places a LaTeX3 "microkernel",

Since I saw on the latex-l list that you're willing to mentor, I suppose
I have no particular objection to adding it to the list. But the
student is going to have to know an awful lot of L3 to have even a vague
chance of getting it done. None of current L3 people are potential
candidates for the student side, right? Will being busy with too many
projects already :).

Implementing float-friendly multicol seems simple by comparison.
But maybe I'm way off track ...

Best,
karl

[Jonathan Fine]

Hello Joseph

Post by Joseph Wright
I've proposed in various places a LaTeX3 "microkernel", and I wonder if
this might be an interesting project for the Summer of Code. The plan
would be to write something based on the existing expl3 code which could
\documentclass{minimal} % Or something similar
\begin{document}
\emph{Hello World!}
\end{document}
without using LaTeX2e at all. A lot of the work would involve finding
how to re-implement the LaTeX kernel in the expl3 syntax.

I'm often asked: When will MathTran support LaTeX? The problem is that
because MathTran uses a secure variant of the plain TeX format (so we
can run TeX as a daemon), and so commands such as \def are stored in
inaccessible locations such as \_def.

Is it possible that this project could be done in a way that produced
something that would help MathTran support LaTeX? As I recall, it took
me about a week to create a secure variant of plain TeX. I'd be really
interested in a similar port of AMS math.

[Will Robertson]

Post by Jonathan Fine
I'm often asked: When will MathTran support LaTeX? The problem is
that because MathTran uses a secure variant of the plain TeX format
(so we can run TeX as a daemon), and so commands such as \def are
stored in inaccessible locations such as \_def.
Is it possible that this project could be done in a way that
produced something that would help MathTran support LaTeX? As I
recall, it took me about a week to create a secure variant of plain
TeX. I'd be really interested in a similar port of AMS math.

I think these are separate projects. amsmath and LaTeX3 have nothing
to do with each other, at this stage. (Although I'm sure we'll need to
write a compatibility layer for amsmath at some stage.)

What are the obstacles to writing a secure LaTeX for MathTran?

Will

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2415 bytes
Desc: not available
Url : http://tug.org/pipermail/summer-of-code/attachments/20090308/6ccae276/attachment-0001.bin

[Jonathan Fine]

Hello Will

Post by Will Robertson
What are the obstacles to writing a secure LaTeX for MathTran?

Thank you for your interest.

It would help if the LaTeX3 project adopted a programming environment
that could produce macros for both the secure and traditional environments.

[Will Robertson]

Post by Jonathan Fine

Post by Will Robertson
What are the obstacles to writing a secure LaTeX for MathTran?

Thank you for your interest.
It would help if the LaTeX3 project adopted a programming
environment that could produce macros for both the secure and
traditional environments.

The current expl3 is set up to provide this already. There is an
option when l3names is processed to remove all of the old TeX
primitives from the user namespace; this automatically makes any
documents processed "secure", since users can't change catcodes and
the only commands provided to them are "document-level" macros.

However, this to be useful requires all the rest of LaTeX3 to be
organised, and that's what this microkernel project is all about
getting started with. We're not talking about actually being able to
write proper documents in LaTeX3 yet, just to be able to create the
format and write "hello world".

To be absolutely clear here: there is *no* user interface to LaTeX3
yet. All of our work recently has been in the programming subsystem
that is intended to also be used for LaTeX2e packages.

* * *

Back to my question: what are the obstacles in providing a "secure
LaTeX2e" for MathTran? I'm guessing you'd need to create your own fork
of the LaTeX kernel plus the packages you want to support [1], and
then re-implement them with primitives that aren't in the user
namespace. Would this be enough work for a GSoC project?

[1]: I'm guessing the main use of MathTran is to provide snippets of
math, so all you'd need to support is the kernel and the amsmath
package perhaps with a couple of font packages as well.

Cheers,
Will

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2415 bytes
Desc: not available
Url : http://tug.org/pipermail/summer-of-code/attachments/20090308/04dc470a/attachment.bin

[Joseph Wright]

The current expl3 is set up to provide this already. There is an option
when l3names is processed to remove all of the old TeX primitives from
the user namespace; this automatically makes any documents processed
"secure", since users can't change catcodes and the only commands
provided to them are "document-level" macros.

What about \ExplSyntaxOn?

However, this to be useful requires all the rest of LaTeX3 to be
organised, and that's what this microkernel project is all about getting
started with. We're not talking about actually being able to write
proper documents in LaTeX3 yet, just to be able to create the format and
write "hello world".
To be absolutely clear here: there is *no* user interface to LaTeX3 yet.
All of our work recently has been in the programming subsystem that is
intended to also be used for LaTeX2e packages.

My idea would be that a microkernel would start with things which seem
most likely to survive (in concept, if not in implementation) from
LaTeX2e to LaTeX3. So concepts-wise:
- Some kind of document class idea (pre-set layout: probably *not*
extra functionality)
- Some division between a header and a body
- Some "end of body" command to mirror the "start of body" command
- Some basic formatting macros
would be somewhere to start.

As a work-in-progress, I'd envisage not changing the LaTeX2e interface
in the first pass. Hence my "Hello World" document example looks exactly
like a LaTeX2e one.

Depending on time, I'd hope that a successful microkernel project would
then look at some more extended concepts (although probably sticking to
LaTeX2e syntax):
- Some basic document divisions
- Some basic lists (enumeration, itemisation, ...)
- The concept of environments
with the target of reversing "l3in2e" to "l2ein3". This would (I think)
be an interesting coding challenge for a student and would also be a
major contribution to LaTeX3 and therefore to TeX in general.

[Will Robertson]

Post by Joseph Wright

The current expl3 is set up to provide this already. There is an option
when l3names is processed to remove all of the old TeX primitives from
the user namespace; this automatically makes any documents processed
"secure", since users can't change catcodes and the only commands
provided to them are "document-level" macros.

What about \ExplSyntaxOn?

Oh, er, if you were in a "safe" mode then this would be disabled :)

Post by Joseph Wright
My idea would be that a microkernel would start with things which seem
most likely to survive (in concept, if not in implementation) from
LaTeX2e to LaTeX3.

I agree with all the comments that you make. My only concern is that
there's lots of "invisible" stuff in LaTeX2e that would need porting
before these high level concepts would be possible (taking up most of
the time of the project), such as the NFSS and file handling and so
on. But maybe some of this could be pasted straight in from the 2e
kernel, to be selectively re-written as necessary by expl3 code?

W




-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2415 bytes
Desc: not available
Url : http://tug.org/pipermail/summer-of-code/attachments/20090308/347420d8/attachment.bin

[Joseph Wright]

Post by Will Robertson
I agree with all the comments that you make. My only concern is that
there's lots of "invisible" stuff in LaTeX2e

All part of the challenge :-)

Post by Will Robertson
that would need porting
before these high level concepts would be possible (taking up most of
the time of the project), such as the NFSS and file handling and so on.
But maybe some of this could be pasted straight in from the 2e kernel,
to be selectively re-written as necessary by expl3 code?

Seriously, I wouldn't envisage starting by trying to implement all of
that sort of thing. The reason my example document was so short was
that it would enable the student to get something working even without
tackling these awkward issues. Getting \emph working doesn't need the
NFSS if you don't provide any other font changing commands! I'm
imagining an iterative process: first get some very basic stuff then,
then look at the next phase, etc.

This is all rather theoretical unless there is a taker, of course.

[Jonathan Fine]

Post by Jonathan Fine
It would help if the LaTeX3 project adopted a programming environment
that could produce macros for both the secure and traditional
environments.

The current expl3 is set up to provide this already. There is an option
when l3names is processed to remove all of the old TeX primitives from
the user namespace; this automatically makes any documents processed
"secure", since users can't change catcodes and the only commands
provided to them are "document-level" macros.

Yes, I already knew about this option. But neither the traditional nor
the secure environments use \cs_set_eq:NN as the name for \let, which is
what follows from your suggestion.

<snip>

Back to my question: what are the obstacles in providing a "secure
LaTeX2e" for MathTran? I'm guessing you'd need to create your own fork
of the LaTeX kernel plus the packages you want to support [1], and then
re-implement them with primitives that aren't in the user namespace.
Would this be enough work for a GSoC project?

I am reluctant to 'fork' LaTeX. I'd much rather work with the LaTeX3
project to produce a secure variant of the relevant style and core
files. Are you up for that?

[Will Robertson]

Post by Jonathan Fine

Post by Will Robertson

Post by Jonathan Fine
It would help if the LaTeX3 project adopted a programming
environment that could produce macros for both the secure and
traditional environments.

The current expl3 is set up to provide this already. There is an
option when l3names is processed to remove all of the old TeX
primitives from the user namespace; this automatically makes any
documents processed "secure", since users can't change catcodes and
the only commands provided to them are "document-level" macros.

Yes, I already knew about this option. But neither the traditional
nor the secure environments use \cs_set_eq:NN as the name for \let,
which is what follows from your suggestion.

Huh? I might have completely misunderstood you. Isn't the "secure"
variant of Plain a specially written variant of Plain that prevents
users from using \def, \let, \catcode, and so on? What's that got to
do with the name of \cs_set_eq:NN besides the fact that it's no longer
in the user namespace? The kernel early on uses \cs_set_eq:NwN as the
literal primitive \let, but later defines a wrapper around it called
\cs_set_eq:NN which is used in most of the other expl3 modules.

Post by Jonathan Fine
I am reluctant to 'fork' LaTeX. I'd much rather work with the
LaTeX3 project to produce a secure variant of the relevant style and
core files. Are you up for that?

In the long run, yes, I think we're talking on the same wavelength.
But if you want to support LaTeX in MathTran in the near future then
LaTeX3 isn't the place to look (since backwards compatibility is only
a long term goal). However, I do hope that the eventual design of
(some of) the LaTeX document classes or document types allow a secure
mode that would facilitate the type of processing the MathTran does.

Regarding forking LaTeX: I would encourage you to do so if the
manpower is available for MathTran; the core and packages you would
support are obviously very stable, and if you rename the resulting
format then there's no licensing problem at all.

Will

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2415 bytes
Desc: not available
Url : http://tug.org/pipermail/summer-of-code/attachments/20090309/f4f2eef2/attachment.bin

[Jonathan Fine]

Hello Will

Post by Will Robertson

Post by Jonathan Fine
Yes, I already knew about this option. But neither the traditional
nor the secure environments use \cs_set_eq:NN as the name for \let,
which is what follows from your suggestion.

Huh? I might have completely misunderstood you. Isn't the "secure"
variant of Plain a specially written variant of Plain that prevents
users from using \def, \let, \catcode, and so on?

Yes. And it stores these primitives in \_def, \_let, \_catcode etc.

Post by Will Robertson
What's that got to do
with the name of \cs_set_eq:NN besides the fact that it's no longer in
the user namespace?

Nothing, except that it uses \_def, \_let etc. And LaTeX2e uses \def,
\let, etc. Sorry, but I'm not yet willing to adopt the expl3 naming
conventions.

Post by Will Robertson
The kernel early on uses \cs_set_eq:NwN as the
literal primitive \let, but later defines a wrapper around it called
\cs_set_eq:NN which is used in most of the other expl3 modules.

Thank you for this. I was not previously aware of this detail.

Post by Will Robertson

Post by Jonathan Fine
I am reluctant to 'fork' LaTeX. I'd much rather work with the LaTeX3
project to produce a secure variant of the relevant style and core
files. Are you up for that?

In the long run, yes, I think we're talking on the same wavelength.

That's good.

Post by Will Robertson
But
if you want to support LaTeX in MathTran in the near future then LaTeX3
isn't the place to look (since backwards compatibility is only a long
term goal).

If I create a secure subset of LaTeX that's incompatible with a
subsequently produced LaTeX3 then they'll be a cost to pay. Either the
incompatibility remains (the users pay the cost) or we remove it (the
developers pay the cost).

Post by Will Robertson
However, I do hope that the eventual design of (some of) the
LaTeX document classes or document types allow a secure mode that would
facilitate the type of processing the MathTran does.

Yes, I very much hope that also. And sooner rather than later. Hence
my contributions to this thread.

Post by Will Robertson
Regarding forking LaTeX: I would encourage you to do so if the manpower
is available for MathTran; the core and packages you would support are
obviously very stable, and if you rename the resulting format then
there's no licensing problem at all.

Thank you for this. However, I am still reluctant. MathTran resources,
like those for LaTeX3, are limited.

[Will Robertson]

Post by Jonathan Fine

Post by Will Robertson
But if you want to support LaTeX in MathTran in the near future
then LaTeX3 isn't the place to look (since backwards compatibility
is only a long term goal).

If I create a secure subset of LaTeX that's incompatible with a
subsequently produced LaTeX3 then they'll be a cost to pay. Either
the incompatibility remains (the users pay the cost) or we remove it
(the developers pay the cost).

In this case I think you're sacrificing too much for a savings that
might not even be realised. If I asked you six months ago the same
questions (before there was any real work being done in the LaTeX3
code), would your answer have been the same? What if LaTeX3 never
ships? (Playing devil's advocate.)

Either way, LaTeX2e will be around for a *long time*. If LaTeX support
is often requested at MathTran HQ, I'd consider forgetting about
LaTeX3 until it appears.

Post by Jonathan Fine
MathTran resources, like those for LaTeX3, are limited.

Indeed! Hence the GSoC mailing list :)

Will


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2415 bytes
Desc: not available
Url : http://tug.org/pipermail/summer-of-code/attachments/20090309/5653f277/attachment.bin