This is too much a philosophical question ;)

In my opinion,the answer may have two faces, for the laymen users we should hide details and make it easily to communicate with computers,

the other face for computer professionals, in this face we should enable of knowledge of all SW and HW details aiming the next forward step.

I don't think productivity is measured through layers of abstraction, or anything like this. In practice, productivity is measured via either LOC (Lines Of Code written), or more to the point, by the number of Use Cases completed in a given time-window. Even in academia, adding more and more indirect references (which is what the "layers of abstraction" eventually amounts to) is recognized that it usually hurts performance, without always improving code re-usability, and certainly, does not "save lines of code written", in fact, adding more layers of abstraction results in bigger programs, that you simply hope that some of its modules will be more "reusable" in the future... of course, if you already have a library that you can reuse for your current needs, then yes, it will improve productivity to re-use it via some wrapping technique or whatever, since it will very likely result in a higher rate of completed Use-Cases per day (or month, or whatever). But one should always be very careful when reusing existing code libraries from other projects, as there are many Dilbert jokes that demonstrate the dangers of improperly using a poorly understood API...

Despite the philosophical nature of this question, the useful answer is no. It will serve no purpose to hide the complexity of computer systems behind layers of abstraction whether you are in the education or business. A sound knowledge of computer systems usually triggers that spark of enquiries that leads and sustains creativity and innovation necessary for growth and development. In todays technology driven society what education practitioners, programmers and the world of work should be focussing on is unpacking and simplifying the complexity of computer systems and making the knowledge accessible to all so that individualised solutions could be developed rather than engage in hiding information behind layers of abstraction. The use of object oriented programming (OOP) and multimedia programming languages like JAVA and ALICE are examples that could contribute to this process. Reuse of codes and integration issues are areas that are worth exploring in detail.

Does someone need to know how an automobile engine works to drive a car? No. Does a mechanic have to have a degree in mechanical engineering to repair a car? Again, no. Computer systems, like cars (and other technologies), have layers of abstraction that are relevant to a particular user -- and even the same user may have relationships across multiple perspectives (e.g., an automotive designer who likes to change the oil of his car and then drive to the store). The vast majority of computer users have no idea how it works -- and they are quite successful. The question here may be better focused on the levels of abstraction that drives hardware and software construction.

In an academic setting, I see abstraction as a means to make it easier to conceptualize and explain the science within software, where different layers of abstraction cater to different audiences.

Laymen, end users (and dead-end users), policy makers, scientists, scientist programmers, and IT professionals are all using our open source software; and each category has profoundly different scope and needs. Although the knowledge in our tools is always readily available, it serves no purpose to obfuscate users with details they are not interested in.

I see abstraction as a conceptual clean-up exercise, not a benchmark for productivity

In my opinion, the answer is yes. I think that the important point here is to have optimized enough abstraction layers in order to achieve good levels of performance.

There is no point in keeping things unnecessarily complicated.

Abstraction in computer systems, assuming that you refer to the concept of abstract types/classes or abstract interfaces are meant to simplify and organize not hide complexity. When computer systems reaches a certain level of complexity these concepts are necessary in order for us to be able to discuss software design in intelligible terms and support orthogonal and collaborative software development. What you refer to as "higher levels of abstraction" has to this date been restricted largely to the industrial sector, simply because academics are usually not good enough programmers to understand these concepts—they have focused on solving problems not creating computer systems that solve problems for them. Science today involves increasingly complex problems and it is my belief that the academic programmer need to learn these concepts and make proficient use of them, or perish.

I do agree with Rodriogo Santamaria, technology should never be hidden. We can think about "critical systems" where abstraction layers are important (no one can control a big jet without electronics/computing/avionics systems - "lots of abstractions"), but you should have several special devices that make possible to audit, to "enter inside the system", as for example, recording devices such as the black-boxes. Layers of abstraction are Ok, but you should think also about: how to verify/test the system, how to garantee quality, reliability, resilience, in this system? Most of the time, low level access to the system is also required and should be carefully designed.

I liked both James' and Dave's replies, as they are both legitimate point of views that avoid reaching the extremes.

I developed software packages to acquire data, implementing from the hardware interaction to the high-level GUI.

My opinion is that software must be designed as a productive tool, so that it is most powerful as it makes things easier for the user. Often this "getting things done easier" goes through simplification, sometimes it does not.

I finally went into implementing straightforward software interfaces for the newbies and the common, more frequent tasks, but as soon as the user was more skilled, it was possible for him to access deeper details of the whole system. All from within a single package.

This required different levels of abstraction, of course.

Now that I work in a company I see that this approach is not followed at all, and that is a pity: programmers want software specs, but they do not dive into the technicalities of the whole package, they want to implement the solution you already solved for them, and when interaction issues among different modules arise it is not their fault...

I know, programming like I propose is not for the faint of heart, but is not productivity an issue here?

Dave, I do remember when cars started to implement electronic devices and all you had to do to check them up was to connect a black box to them. Seldom the results of the check-up were simply WRONG, and a good technician could easily spot what the REAL problem was.

So, to wrap up the discussion, layers of abstraction are good when they make sense. But let's keep in mind that we are developing tools for somebody else, not for our sake!

i fully agree with Hans Elmlund - abstract layers are mostly about simplicity and code organization and discipline not about hiding complexity. My experience says some lack of the three (simplicity, etc.) is not unusual for academic programmers..

All,

Very interesting answers, they will give me things to ponder about, thank you. Some of the additional concerns based on your feedback are:

a) While I understand the car analogy, the concern is that this is the engine that manipulates the information that we use to extract knowledge. Information compromise and information manipulation is so widespread as opposed to car engine manipulation (most of the time put in a cheap copy of a car part engine and it just wont work even if it says it replaces the original :-).

b) My second concern is the misinterpretation on the side of the user when he uses canned programs. My experience is that the dependency of the results is almost worship like and they do not stop to think about the results. The same goes for programmers. I have experiences similar to Marino's in that programmer want just the software specs not the technicalities and end up with the wrong results.

Can these problems be addressed in an efficient manner ?

In my opinion, abstraction is not a way to hide complexity but rather a way to overcome it when we need to reason or understand systems. Engineering proceeds (also in computer systems or software systems) by disregarding aspects which are not relevant for the intended purpose whilst other aspects are highlighted in order to better handling them. So which parts of a system we do not want to see depends on the application at hand. But of course, having the possibility to get deep into the ultimate details and structures is crucial to have full control and knowledge of things. Abstraction is a way to move up and down there...

With the advent of CLOUD Computing(Common, Location independent, Online, Utility services, on Demand -according to At & T) every thing in Computation is hidden behind unknown layers,with IAAS,PAS and SAS Provisions.

P.S.

We cannot graduate only computer engineers much like we cannot graduate only programmers. It always bothered me that, students from the CS departments are less aware of the architectural components, and they would like the abstraction layers increased. ECE students were always closer to the hardware. The question is : assume for a second , theoretically, that, we have three textboxes, asking you three questions, and a button that says "GENERATE PROGRAM" and the programmer can generate the program with 100x less effort. This will be revolutionary.

So, this will require major advances in Artificial Intelligence (AI), and will allow everyone to be more productive. The big question is : who will eventually understand the deep-down details to be able to 1) connect AI to Computer Architecture, 2) write the software which will require deep understanding of the hardware, 3) eventually compile the program, which requires deep understanding of hardware ...

Mu summary : We need both. It is Ok to significantly increase the abstraction layers and make programming easier. However, the more you increase the abstraction layers, the more people you will need working in the "lower layers." And, this increase in abstraction levels is limited by PERFORMANCE, since performance vs. abstraction levels are inversely proportional. IF this wasn't the case, C (or C++) would disappear by now, and Python (or Java) would take over. It didn't. Because, a programmer should always have a reasonably good understanding of the hardware to write high-performance code.

Tolga,

I usually get complaints from students complaining on why do they need assembly and computer architecture classes and not be programming with higher level languages solving practical problems. I think that you are right that we need a balance but how do we achieve it with so many layers. It is so difficult to strike a balance in which everyone is happy!

Me too !

My analogy is that : Learning more about architecture is like a bad tasting medicine that is GOOD FOR YOU ! You have to take it ! You will get better :)

Also, teaching is like parenting. If you allowed your kid to do everything they LIKED, you'd be in trouble ! Sometimes you have to give them what is good for them, even at the expense of being not-so-popular now, and they will appreciate it later. I have students that write to me and say "I am so glad that, you taught us the fundamentals." But, of course, some of them will get turned off to the subject, and will never come back to it. So, Arturo, you are right ... It is hard to find the right balance. Who said teaching was easy anyway :)

While most of the people in this thread agree that abstraction is not equivalent to complexity hiding. It has been my experience from OOP that abstraction coupled with encapsulation handled incorrectly produces the hiding of information especially in commercial environments.The information hiding has been so bad in some of the projects I have worked in the private sector that it lead to immense code bloating.

Maybe I am wrong, but that has been my experience.

1. "abstractions layers" are often more complex (and less well-documented) than

more concrete "models".

2. There are reasons D. E. Knuth uses MIX in his books.

3. "productivity" probably really means how solid the code is, and how easy

it is to maintain (i.e, how well-documented.) Not either how fast it is or how

long the coder took to get there.

4. See www.civilized.com/programming.html (How to write computer programs)

Dear Gary, if you know of an organization where the management measures "productivity" by how solid the code is and how easy it is to maintain (by who?), then please tell me so I can send them immediately my resume!!! I have worked for large and small organizations, pure commercial ones, and more research-oriented, in "industry" as well as "academia", and I have never found a manager whose metric of productivity was what you define it to be. Productivity is always measured as "work done per hour" and in the software business, "work" is usually measured by Use-Cases completed. Anyway, (unfortunately) this has been my 20+ year experience...

Modern systems are categorized as complex system , so naturally to have layers of abstractions to reduce complicity and to have user friendly interfaces

Gary, I would like to add one more thing : 1) SCALABILITY, which , I think , is different than 2) MAINTENANCE.

SCALABILITY=Being able to expand the code and add a lot more functionality ...

MAINTENANCE=Being able to make slight improvements to the existing code and remove bugs, account for conditions that were not originally planned ...

Going back to the original question, COMPLEXITY HIDING should help both ... As long as the code is written well, the performance should be good too, even if the complexity is hidden. C++ , I guess, is the perfect example of this.

Even Java is good in that sense. What makes Java lower performance is the additional translation layer (JVM). So, I guess, it makes sense to discuss a language in terms of its runtime environment too. When you says "Java," you really cannot separate it from JVM, they are both a part of the same thing.

I keep going back to PERFORMANCE, because, most of the times, this is important ...

Knowledge of Computer Architecture, Instruction Sets, Registers and Data Management, Addressing Modes, Interfacing and all are the Building Blocks for Good and Efficient Programming.

Even if you go for Cloud Computing also, you place your Indents on Cloud Providers based on these Criteria.

Additional important requirements are the Formation of Logic and Algorithm, and a good Knowledge of UML. These are the things that differentiate a Software Expert from Programmers.

Software Testing is another important area.

Good Domain Knowledge helps you in Developing Application S / W Packages in your area.

P.S.

Totally agree with Rodrigo.

It is a matter of perception, nonetheless abstraction layers do exist, and they are fundamental to help us for better understanding and being aware of the problems we are facing.

I think that the perception of some "productive programmers" (and even some clients of that branch), is that there is no other thing, than what they know or what they can see applied and that the "faster results imply better quality". I mean an empirical viewpoint of knowledge, and I think that the scientific method and systematic approach are beyond that, as Descartes said, they are required by the daily life, but today many people do not think that way.

Jeje "hidding or not the complexity" is a very complex question, because, who defines what is complex or not, or who defines the reference point to put a value to something, just because most of the people do it, does not become in the purely truth I think; but money rules this world.

So I think that if I support the power of the knowledge and the sharpness of the scientific method in combination with basic science topics, (maths above all, as a perfect language to describe things in a short way and precisely), then:

*I could apply them to get a faster developing process of something (software in this case), covering the "need of fast results".

*I could apply them to convey a more specific knowledge for those who need it, to teach how wide is the universe of information.

*I could apply them to build my own hierarchy tree, taking what I need when I need it from each layer as Rodrigo said.

* And finally I can apply them to show, as an human being, an example of what the scientific approach has to offer and to improve.

Thats my opinion, very short and subjective, it implies alot of things, good and bad, but I have wrote it with a constructive purpose.

Cheers.

Karosuo

There's no computer science problem you can't solve with another layer of abstraction. Except performance.

So the answer depends what you are trying to do.

If you are coding a hard real-time system that fails if it can't respond in time, you care more about performance than if you are designing a little widget that most people will use once then move on.

Dear Arturo,

I don't think that abstraction does very much service when it hides complexity. Often enough I find that abstraction generates its own set of demands without fully protecting one from the realities one was avoiding. So if I want to draw a graphic, I can do so rapidly in a low level language but a higher level language, which handles the general case of many devices and many possible graphics, will impose both overhead on my learning curve and on my code. Similarly I can store my array as a list of .txt entries or use a database.... Obviously, the correct tool makes a match between both the programmer and the situation!

Now, can you find a useful, general, (abstract?) definition of productivity? No. Just specific ones that describe our needs in a particular situation. I certainly appreciate the effort video game engine designers put into optimization of critical routines. I have not had to do this for years; although I was very good at building real time control systems it is a mismatch for me to apply these skills to the work I do now.

I think that productivity as the time saved by a programmer is equally problematic. I wish I knew, before I embarked on something, which approach would be most efficient! In some cases we can show that abstraction is definitely not making people more efficient...

To return to my opening thesis - complexity that one hides almost always comes back. Usually through fragile designs that cannot stand small changes in designed function or output. I am happy enough to use compilers to distribute my work over threads, cores, processors, and machines. But pretending that this can be done simply because my language standard permits it is a recipe for disaster. I seek abstract, general solutions not ignorant ones.

Good luck

Abstraction is a microscope that lets you zoom into your level of utility or interest. Thus, it is a useful mechanism. E.g. If you are working with a communication protocol you do not want to know the details of the hardware interrupts and vice-versa.

In the real world, you highlight a philosophical divide. Industrial perspective of make it work (or make as many things work in given time period) versus the academic perspective of computational efficiency. What is more important? Depending on the usage of system (and chance) one of these philosophies seems to be more applicable.

1- Contractors who supervise the programmers in general do not understand inefficiency that cannot be demonstrated. Their contracts are based on "cases"; the computational efficiency typically is not in the equation. Since money drives the development process, the accepted definition is: Efficiency = Features coded/ man hours.

2- The feature that you code might be executed rarely.

3- The scaling of the system is not done to the maximum where the inefficiency effects QoS measurably. Further, at a given moment, the inefficient code might be executed in a few threads only.

However, there are instances where systems were redesigned because of slow response from soft-ware component of the system. Further, unless the issue is serious, the software computational inefficiency is not detected (and localized to a particular piece of code). Thus, people tend to live with an inefficient system oblivious to the cause of inefficiency.

Your point about efficiency is definitely valid. Just using more computational power to overcome ones inefficiency will not always work. Most of the computationally hard problems are solved by efficient algorithms; not even the fastest of the computers can solve these problems even for a moderate sized input.

I agree that some academic/semi-academic organizations do have efficiency requirements. However like Ioannis points out this is not common.

In real life people complain about a system getting very slow because it is old!! If you look into it you may find that the system is scaled up and the sloppy code is being executed in several threads. However, unless you are an expert who would know this. Further before such a thing happens, typically systems are replaced. This hides the inherent sloppiness!

Good discussion, I hope that efficient coders are well recognized. Best regards.

Dear Bhadrachalam, also bear in mind that productivity measured as "Use Cases completed per time-quantum" in no way means "sloppy, slow code implementing a feature on a GUI completed per time-quantum", although in some cases it might well be. For example, if you work in a semi-academic setting (e.g. Bell Labs etc.), the "Use Cases" to be implemented may well have very demanding performance related non-functional user requirements associated with them, e.g. "Requirement 1324: The system shall complete all monthly schedules of the pilots of the company within less than 5 hours of wall-clock time on a RISK R6000 machine running AUX". Such requirements associated with a particular Use-Case of the system detailing the construction of the monthly schedules of the company's pilots, essentially imposes a very serious demand to write the most efficient code for the particular use-case. In this example, the Use-Case would not be considered complete unless the system ran in less than 5 hours, thus the definition "work completed/time" can very easily imply very strong efficiency guarantees. In other cases (usually in smaller-scale projects e.g. small business-related web-apps), it is indeed the case that the efficiency of the code, or the readability / maintainability of the code is partially or completely ignored in favor of "delivering a working version".

And a final thought on the issue of "layers of abstraction": unless your "layers of abstraction" serve some useful purpose other than what *you* perceive as "good code", then the whole issue becomes an exercise in aesthetics, to paraphrase a well-known quote from the fathers of RISC computer architecture.

Ioannis,

You have a good point regarding non functional requirements and I wish PMs would pay more attention to them (I would call your example one of the few exceptions from my experience but I may be wrong). The sad part is that when budgets are tight this is one of the first things that go out the window, unless something really bad happens as Bhadrachalam pointed out.

Excellent discussion, thank you very much for the the comments.

I'd like to share with you a text I wrote some while ago:

We live in a society of software-predominant systems. Computer systems are everywhere around us—from

supercomputers to tiny embedded systems—and we all know how important it is that services appointed to

computers be reliable, safe, secure, and flexible. What is often overlooked by many is the fact that most of the

logics behind those services, which support and sustain our societies, is in the software layers. Software has become

the point of accumulation of a large amount of complexity. Software is ubiquitous, it is mobile, it has pervaded all

aspects of our lives. Even more than that, the role appointed to software has become crucial, as it is ever more

often deployed so as to fulfill mission critical tasks. How was this possible in such a short time? The key ingredient

to achieve this change in complexity and role has been the conception of tools to manage the structure of software

so as to divide and conquer its complexity.

• Dividing complexity was achieved through specialization, by partitioning it into system layers (for instance

the OS, the middleware, the network, the application, and so forth).

• Conquering that complexity was mainly reached by hiding it by means of clever organizations (for instance

through object orientation and, more recently, aspect and service orientation).

Unfortunately, though made transparent, still this complexity is part of the overall system being developed. As

a result, we have been given tools to compose complex software-intensive systems in a relatively short amount of

time, but the resulting systems are often entities whose structure is unknown and are likely to be inefficient and

even error-prone.

More on this in the following papers:

https://dl.dropboxusercontent.com/u/67040428/Articles/adidrds.pdf

https://dl.dropboxusercontent.com/u/67040428/Articles/ads7_deflorio.pdf

Vincenzo,

Thanks for sharing with us those thoughts.

To continue this discussion: If there is the linear growth of defects per lines of code, and we have the modularity to achieve exponential growth(assumption) and this makes it easier to develop, then why do we have a linear growth in bugs?

As per your post which I think is very interesting an true "but the resulting systems are often entities whose structure is unknown and are likely to be inefficient and

even error-prone"

What is this underlying trend attributable to?

One of my thoughts is that modularity itself and hiding the complexity is at the root of this phenomena. For example, in layering systems(take TCP/IP or the division of information systems into programming and systems management, etc.) there are gaps which are assumed are covered by the other layer and it ends up not being covered at all. Also, there is "impedance" matching to be made which ends up in inefficient code implementation. Further, cross cutting concerns made for inefficient code.This is one of the main reasons aspect oriented programming came to be(which is also lost in this layering if done by independent parties). But this in turns make other problems such as spaghetti code. This just substitute one problem for the other (while I do appreciate Kiczales contribution, I think this is a shortcoming of aspect oriented programming).

Thoughts and criticism are welcome.

Hello Arturo,

thanks for your valuable thoughts. First of all, you say "why do we have a linear growth of defects"; I'm not certain this growth is actually linear. Defects in software are very difficult to identify, let alone quantify; the problem is exacerbated from the fact that defects may stay latent and only manifest when certain conditions manifest themselves.

A second thought is that a "new software development" is required; one in which the decision of what to conceal and what to make evident may be revised at software composition and deployment times. In the second paper I sent you I discussed this issue. I believe a way to soften the problems we are discussing here is by having software components expose in a standard way their design, deployment, and run-time assumptions.

Citing again from my paper,

"Across the system layers, a complex and at times obscure “web” of software

machines is being executed concurrently by our computers. Their mutual depen-

dencies determine the quality of the match of our software with its deployment

platform(s) and run-time environment(s) and, consequently, their performance,

cost, and in general their quality of service and experience. At our behest or oth-

erwise, a huge variety of design assumptions is continuously matched with the

truth of the current conditions. A hardware component assumed to be available;

an expected feature in an OSGi bundle or in a web browser platform; a memory

management policy supported by a mobile platform, or ranges of operational

conditions taken for granted at all times—all are but assumptions and all have

a dynamically varying truth value."

"Our position is that existing tools will have to be augmented so as to min-

imize the risks of assumption failures e.g. when porting, deploying, or moving

software to a new machine. We envision novel autonomic run-time executives

that continuously verify those hypotheses and assumptions by matching them

with endogenous knowledge deducted from the processing subsystems as well as

exogenous knowledge derived from their execution and physical environments.

Mechanisms for propagating such knowledge through all stages of software de-

velopment would allow the chances of assumptions failures to be considerably re-

duced. The ultimate result we envisage is the ability to express truly assumption

failure-tolerant software systems, i.e., software systems that endorse provisions

to efficiently and effectively tackle—to some agreed upon extent—the problem

of assumption failures."

In that paper I identify three hazards of software development: the Horning

syndrome, the Hidden Intelligence syndrome, and the Boulding syndrome,

which I exemplify with well known examples of software failures.

I also introduce the concept of assumption failure-tolerant software

systems and provide three examples of strategies—one for each of the mentioned syndromes.

I'll be glad to continue this discussion and provide you with further information if deemed useful.

This is a quite interesting discussion that relates deeply with foundation of computing. Abstraction is the technique commonly used for giving to user an interface to low level complexities. With abstraction operative systems give to users the possibility to interact with hardware without having to handle devices by hand. With abstraction object oriented programming aims to enhance re-usability of code by hiding low level low level implementation by implementing an API.

We can define abstraction as a "language" that is put on top of a computing system for handling in a higher level of human understanding the processing complexity. Of course people from industry want to have a "language" having both characteristics: Simplicity (easy to learn/use) and Completeness (to take the most of your system). However, as long a "language" is complete it seems equivalent to any other language. This fact is claimed by the well known Church-Turing thesis. And as long your language is "complete", it also will suffer from uncomputable problems. For example finding in any "program" in your complete language a bug that perform an infinite cycle is impossible. This barrier turns your very "abstracted" but complete abstraction in yet another language with buggy and unproductive programs.

If you think that completness is the problem, then you are righ. In a industrial setting economic equilibrium more or less approximate, that means at some point each company will have the same incomplete abstraction that resolve the business problem. Furthermore, winning companies will tend to improve theirs programming api for completeness in-order to to win the market. So completeness is unavoidable for companies.

The previous argument is my believing that the dream of a highly productive computing system is cursed to remain as complex as many others. I believe this is a natural law that constrains computing systems. I believe the only way to circumvent this issue is by hiring very skilled people for building and maintaining your system. The worst will be to replace skills by system abstraction.

In spite of "religious programming language war", better abstraction will always help humans to work with computers. Nevertheless, in this area the main problem is not from the mathematical-computational domain. Nevertheless, the best abstraction is related to Psychology factors and domain related. I never seen before a Psychology-based programming language that commit an "optimal criteria" for easy usage/learning. Most programming language are invented by computer people, and the infinite discussion of which language is the best continue without firm argumentation.

every computer system is an abstraction, beginning at the hardware level (eg. CPU-s are just an abstraction of some concepts on computation we have). I think the decision of what layer of abstraction one uses mainly a function of the required functionality. If I wrtie a program for testing an idea , i can do it in the lowest layer. At the very first moment I want to share the results, or want that computaton to attach to a specific system, I must use some of the abstractions of that system. (eg. if my program have to communicate through a LAN, I have to use the abstractions created by that systems designer (802.x frames and the like for example). The complexity may, or may not be hidden, depending on the context. Every layer of abstraction is a model of the real thing, based on some assumption or agreement of what parameters of that complexity can be thougth to be non-important. I for myself tend to use the lowest possible layer that does the job, mainly because every layer at least doubles the TOTAL complexity of the system.

As a sidenote, I found Vincenzo Florio's comments very much like my own perceptions. We live in a very complex world, and I think we want to make it more understandable. Unfortunately by creating these abstractions and then treating them as entities of reality we may add to the complexyti of the system, a situation that we try to solve by ading yet another abstraction which of course we again forget to handle as abstractions and so on...

A system can have several abstraction layers whereby different meanings and amounts of detail are exposed to the programmer. For example, low-level abstraction layers expose details of the computer hardware where the program is run, while high-level layers deal with the business logic of the program. In object-orientation, abstraction determines the essential characteristics of an object. It captures only those details about an object that are relevant to the current perspective. There exist several levels of abstraction in a typical object-oriented system. At the highest level, a program is viewed as a “community” of objects that interact with each other in order to achieve their common goal. The important features of this level are the lines of communication and cooperation, and the way in which the members must interact with each other. The next level of abstraction is neither found in all object-oriented programs, nor is supported by all object-oriented languages. However, many languages permit a group of objects working together to be combined into a unit. Examples of this idea are included in namespaces in C++, packages in Java, or units in Delphi. The unit allows certain names to be exposed to the world outside the unit, while other features remain hidden inside the unit. The next two levels of abstraction deal with the interactions between two individual objects. This intuition could be described by communication as an interaction between a client and a server. The next level of abstraction looks at the same boundary but from the server side. This level considers a concrete implementation of an abstract behavior. Finally, the last level of abstraction performs a specific activity, usually defined by a single method of a class. For example, the technique used to perform the addition of the most recent element placed into a stack.

Some levels are closed to the device such as the multiplication of integers. Being closed to the device means that an application is not dependent on many other levels of abstraction. Other applications (e.g., getting GPS maps on a Smartphone) are not closed to the device and are dependent on several other levels of abstraction. There can be a tradeoff between the number of levels of abstraction in a device and the efficiency of an application on the device. Too many levels of abstraction might slow down an application. On the other hand, too few levels of abstraction might result in applications that have low cohesion or take too long to create in markets where being first is important. The potential tradeoff between the number of levels of abstraction and efficiency is one of the many challenging problems in computer science.

I find Mokter's comment the most sensible one.

Reading Vincenzo's idea to embed more knowledge (i.e. complexity) in the software itself so that it could constantly check that the assumptions needed to run a certain routine hold true gave me the shivers.

It is not a matter of any syndrome, but an engineering perspective: the more complex a machine, more parts prone to fault you have, and who will check that part of software that will check that the certain routine can be executed?

I previously wrote myself that embedding knowledge of the system inside the software is good, as it will allow to offer the user different levels of access to the system. But I really fear this knowledge of the software as a dangerous overhead.

And by the way, this will not solve the real issue: what if the control software will find the hypotheses non valid in a critical routine? By the look of it, it should simply not perform the routine (program halts with some error message, I presume), whereas nowadays it would spit out the wrong result or an error message. So I don't find any real improvement in this.

I will make two cases: division by zero. We all know that it should not be made, so we can write an if condition to avoid it (so for any computation-intensive application that would be a curse) and get NO result at all; Challenger disaster. Software was trying to correct the descent by controlling the wings (wrong assumptions in the correction software), eventually accelerating the disaster.

Bottomline: every single piece you add to a complex system is prone to error, even a simple hammer+needle. You can reduce this error as much as you want, but sometimes the hammer hits your finger, no matter what.

How Others already said, things must be done as simple as possible, but not simpler than that. What I propose is to add enough complexity to expose one's assumptions and to have some generic approach to match these assumptions with those of the underlying layers. This complexity would be minimal and I think worth adding. It would allow assumptions to be verified at design, deployment, and run-time. Coupled with diversity it would enable to cope with those run-time faults one has foreseen code to tolerate.

Do we need to add ever more functional complexity in our codes? I think so. I think it is this complexity that is reaching the point of non-manageability and should give us the shivers. Our software is no more as simple as hammer+needle -- we must live with this. Living with this will require some additional complexity -- both architectural and in the application layer.

OK, now I got your point: "things must be done as simple as possible, but not simpler than that" is an excellent perspective.

Thank you Marino.

No, the complexity of computers should not be hidden behind but illuminated by abstraction.

At its essence, a computer is a device which reads, computes, and writes data. If anything we ought to employ the abstractions to fold the complexity of a computer and its state information into human readable interfaces.

Abstraction should not be obstruction, but a tuned metaphor for the underlying mechanics of what has been abstracted. Similarly, one ought to be able to move from any layer of abstraction freely and easily without needing domain expertise to observe state and changes in state as a result of a program.

The above given answers are good

Productivity is a solution for PROFIT.

I propose you define the dimensions of productivity as a function of your bigger goal. For example: Scale-ability (degree of freedom), Life Cycle(time) should be defined before the abstraction and in accordance to your business model.

When you exceed the limits of your definition, you will know that your rule might break. Hence the end of the life cycle, and redefinition of the abstraction based on new goals and new foundation.

MY GOAL: to avoid building layers on nearly obsolete layers which might be built on obsolete layers. Knowledge & Control of these lower layers definitely helps me maximize my profit.

Can you imagine building a website in HTML for a quantum computer? Definitely possible and possibly expensive and maybe useless.

"Productivity through higher levels of abstraction" is what is of importance to a Application User,but, for a Developer Lower Levels of Abstraction are also important because he needs to know the Requirements and their competency.

Obviously not only Optimized Code but also the H/W and S/W and Accuracy Requirements for the Program play their Roles in Faster Execution of the Program.

As a matter of Fact. Real Time Programs need Faster Algorithms,with Reasonable Approximations probably at the expense of some accuracy.

Judgement depends on one's Point of View.

P.S.

Reza, sorry, but profit already proved to be a shortsighted paradigm for many, many applications, even in the financial world, its own realm.

It is by pursiung profit that the whole world is suffering this financial crisis. Makes you think, doesn't it?

Moreover, I don't want my calculator to sum 2+2=5, no matter what abstraction layers it implements, because the correct result was too expensive to be obtained!

Marino, I'm glad you picked up on my subtle provocation; short term goals do often ally with greed, and do diminish long term gains. Long term planning should be present in the business model.

Calculators add 1+1=2. But in reality we know that cooperation yields results that are greater than the sum of individual work. Perhaps we'll optimize our function to reflect this in 'calculations' where 1+1>2 or 1+1

I think that the question needs better context.... :

"While outside the academic sector, a great majority of consultants and programmers cherish the notion of "productivity" through higher levels of abstraction."

Outside the academic sector, everyone has to value productivity. This is not only through "higher levels of abstraction". And, it is not to the exclusion of other metrics which may be just as important - even though some may believe that's axiomatic. It all depends on the objectives. I hasten to add that productivity can be valued within the academic sector as well - wouldn't want to leave anyone out!

"Should "productivity" be measured by the time saved for the programmer or the optimized code underneath? "

I can define productivity in terms of output per unit time (while "time saved" is a more elusive notion - what's the basis for the measurement?). It raises the question: "What is output?". How is output measured? Lines of correct code per unit time? Lines of tested and somehow certifiable code per unit time? etc.

So, productivity need not and should not be viewed in such narrow terms as the question suggests. So, I think this nicely deals with the "or the optimized code underneath" contrasting statement.

"While they are not necessarily exclusive they usually are."

I find this a most pejorative assertion. Take it to the extreme. The code is generated but it doesn't work nearly adequately, is dangerous, etc. Can that happen? Of course. The problem is in the eye of the beholder and there are many, many levels of acceptability that don't include "doesn't work" and "is dangerous".

I don't know if any of the correspondents here consider themselves to be engineers but good engineering starts with some kind of spec's regarding those things.

I find the contrast suggested rather annoying. Once upon a time I launched an internal website using automated tools. I don't pretend to be a great programmer but I needed something that would work. It worked and the result was satisfactory for my needs. I caught a huge ration from one of our "real" programmers who was aghast that the code didn't purely adhere to one of the current standards for html. I argued that it didn't matter. There was never agreement on this tangential subject.

Let me pose a question for the academic community:

Is it axiomatic that abstraction methods lead to errors in code?

If so, why aren't the abstraction methods better? More robust? etc.

That could be a valuable academic exercise.

The label "greed" suggests a fundamental lack of understanding of the give and take pressures that are necessary in getting things done in the real world. It's not greedy to work hard to stay in business and protect the jobs within an organization - it's a necessity. I have worked in many more situations where this was the reality than not. And sloppy work and inferior products was not tolerated.

Abstraction is a good thing, because it reduces the burden on programmers who are constantly getting more expensive, and it allows others who's first skill is not programming to bring their contribution from other disciplines with a lower barrier to entry. And as hardware is constantly getting cheaper and more powerful the execution speed overhead of the abstraction becomes less and may soon be negligible.

@Fred Marshall (and the rest) "Is it axiomatic that abstraction methods lead to errors in code? If so, why aren't the abstraction methods better? More robust? " I think in a way those questions can be addressed through the theorems of Church-Turing and Gödel:

In the Church-Turing case, David Hilbert posed the Entscheidungsproblem, which very succinctly request an algorithm " to decide whether a given statement is provable from the axioms using the rules of logic" (http://en.wikipedia.org/wiki/Entscheidungsproblem), and Church and Turing proved that a *general* solution to such problem is impossible, thus setting some limits to our levels of provable abstraction.

As for Gödel incompleteness theorems, which (very roughly speaking) state that we can not have a system for *all* mathematics, which could be complete and consistent at the same time. "For any such system, there will always be statements about the natural numbers that are true, but that are unprovable within the system. The second incompleteness theorem, an extension of the first, shows that such a system cannot demonstrate its own consistency."(http://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems)

None of the above theorems forbid us to continue research to improve the algorithms we already have. We merely have some boundaries saying somehow that perfection will always be an endless quest, just to put it in philosophical terms, and I do hope that such statement doesn't sounds preposterous or patronizing, because neither was intended.

On the contrary, to me it sounds reassuring: there is still a long job to be done, and as I have job to do, let's get busy! :D

Abstraction is great but a sense of what’s underneath is important too. For example, if you understand how things like dynamic dispatch and recursion work at a low level, you are less likely to misunderstand them at a more abstract level. I recently ran a course that started from MIPS assembly language and worked its way up to C. The class also do OO languages. It will be interesting to see in a year or two if they have a better integrated understanding than students who have not had that approach (though hard to measure, too many variables). You can find the book I am working on linked below that uses that approach.

http://homes.cs.ru.ac.za/philip/Courses/CS2-arch-C/

Conference Paper Teaching Without Technology

@Philip Machanick

Thanks for providing informative link and brief as well as rational arguments. I agree with you.

Hello,

"Abstractions" are seen as results coming from the action of abs-trahere = to pull from.

Animals produce mindings and thoughts from sensorial experiments being translated by transducing and coding up to be driven and stored and computed in an " abstract " shape from  peripherals to nervous centers  ( not only brain ). Related cybernetically correlated actions of effectors guided by goals play a great role.

To begin to think about induction by Artificial Intelligencies and Ontological Biologies we must place them in the situation of experimenting directly our environments so as to be able to teach us what they found.

Please notice that for a mathematician "abstraction" he works about is what is the most fundamental articulation between axioms facts operations and conditions he tries out . Speaking about "conditions" note please that " is transcendantal the most general study about " a priori " conditions and/of functioning of thoughts " this just implying as "a priori " the a priori of stating ( as philosopher and scientifically ) not any a priori at all but a Generalized Relativity Principle where no hierarchical order of any kind preexist but what  results from all relative interactions of all things each to each and each to groups and groups to groups and systems to systems until The Whole is concerned. Imagine please how relative between each hydrogen atom is the commonplace process of interstellar local gas accretion creating a star. Would you please note also that " hierarchy" does not mean at all a pyramidal order but the " sacred command  " where "arch-  " comes from " arkeo "= I push or drive back the pain. To command is to make up ones mind  ( and more ) to get rid of pain troubles and so on... So to command should not be to impose by any means one's self one's ego as it is reduced to and lived from both sides most of the time...

Jean-Yves

I am not a programmer (although learned FORTRAN many years ago, it was very explicit, but modern programs (even "Mathematica") do not reveal algorithms to users) but user. I know something about the economic notion of productivity. My answer is close to the raised question, but is not exactly to it.

I have some feeling that because of existence of billionaires (Bill Gates et al) in Information Technology and Computer Systems too much money is invested there, and users suffer from it. This exponentially growing information volume does not increase their productivity significantly. Some people would also prefer to use the same Word program for 10-20 years, but they are forced to learn new versions quite often. We have too much software invented that many people would not like to use. But they are forced to learn afresh because any new smartphone is using new software too. A lot is just to remember where and what to click; it takes a lot of our human memory.

Next, storage of information becomes cheaper. Some years ago a personal computer with 1 Gb operation memory and 100 Gb storage memory seemed good, but now not. But do we use this memory efficiently? I think not, because new software is using more and more memory.

As for complexity, it is better to reveal it to users. It may happen that they do not need what they are forced to buy. I would keep all old systems used in 1990s as well and give consumers free choice. I think many would be happy if there would be much less programmers today who invent and force to use new and new systems, which give almost zero additional utility to customers, but only the cost of learning. A counter-example is LATEX that exists for maybe more than 30 years without change, and many researcher are happy with it.

I think its necessary to hide complexity in computer systems. Mainly due to the level of knowledge that is out there. A user do not need to know all the complex structure for him to use the system.

I know, I am five years late to the party, but there is a point missing, imo.

I agree with a lot of the above answers stating that abstraction does not (necessarily) mean hiding and that well-defined layers of abstraction are crucial to build complex systems.

But I would like to stress a historic view on this topic in the form of a question, I hope that doesn't come across too rude:

Are you really questioning what has been done in science and engineering (including computer science and software development) over the last centuries?

And to answer your question on how productivity (by which I assume you mean efficiency) is measured. It is both, development time and execution time. Depending on how often and how long the code will run, one or the other is predominant.

I will answer directly your question with a yes. From your more general comment than that of my question, the reason is that there is much to learn by looking at complexity without hiding it (that is why there is such a thing as a science of complexity). To elaborate on why my questioning over conventional wisdom of the last centuries stems from the basic fact that historically we did not have computers so we had to hide complexity and do toy problems in S and T. This also has been carried on forward into computing itself.

A good point made on one of the posts is that abstraction is not necessarily complexity hiding (though in my experience it usually is), though my point is more from the point of view of understanding and problem solving. Solving a problem without looking at the layers of complexity can lead to errors in the solution. But I guess the question is very open in scope and can be interpreted differently based on the ambiguity of the terms.

I understand your point. And I agree partly.

Using black boxes *is* indeed a problem in that the user will not understand the validity, scope, limitations etc. of a solution, if he/she doesn't understand the underlying math or science.

The problem that I see is: where does it stop? An engineer choosing the right material for a construction has to have an understanding of material science and mechanics and continuum mechanics. But does he have to understand the solid state physics which "causes" certain properties? Or the particle physics underneath that? Quantum physics? The math behind that? All of it?

I am a strong defender of , e.g., teaching students numerical simulations by requiring them to derive the differential equations and write their very own small simulation code. Even though there are dozens or hundreds of solutions and great codes available. And yes, after that exercise and probably for the rest of their professional lives, they will use one or several of these codes. But (and in that I agree with you) they have to know the basics.

But I will not teach them basic electronics so that they actually understand what happens in the computer when they run that code. And I am not teaching them how OSs work or compilers (as two examples) for that matter. And in that, I am hiding complexity from them if you will.

And back to the historic context: The advantages of the last centuries have been possible, *because* we take differential calculus, how computer work and whatever the compiler does for granted. And we don't have to understand all of it and we don't need to prove that they work every time. It is just not possible to do modern science or development and not hide a significant amount of complexity from oneself and others.

"Solving a problem without looking at the layers of complexity can lead to errors in the solution. "

Do you mean layers of complexity of the problem? or layers of complexity of the solution? Or, perhaps, layers of complexity of the situation in which the problem is posed?

I was once advised to stay out of the technical details of a situation until the situation demanded it. Before anyone criticizes this advice, let me say that it was just another way of saying: "Don't get mired in the details as a general approach".

Some people do that. I am reminded of the difference in preference styles from Myers-Briggs in comparing the intuitive "N" with the sensor "S". The former tends to start from the big picture and bores into details when needed. The latter starts with all the details. Since I'm an "N", I've always thought the latter approach was a time-waster and often silly. After all, one could engage in a useless pursuit after learning all the possible details vs. deciding if there was a context for further effort.

I have the feeling that there's an "axe to grind" buried in the OP's question and response. But I don't yet get it.....

I think it is a general practice in system design of all kind and not confined to Computer systems only. Why would a common user like to bother about system complexity? Experts will, of course, will like to know and study the complexity issue. Hiding the complexity does not mean one can get away from it. It is a just a practice of not letting the user know who does not need to know.

Hiding complexity from the user is good, but hiding complexity to facilitate the construction at higher levels must be done with care.

Although you asked the question a while ago the topic is more relevant now as hackers exploit the gaps in the system which are there because of the poor understanding of system complexity and trade-offs at the higher levels.

I think this is a reality that we can not deny, but we must work as simplistically as possible to deal with that complex environment. for developers they knows the degree of complexity, but for normal users they maybe know nothing about that , this is a positive thing.

Had to be away for a few days, but to address some of the comments above i will say that in my experience the devil is in the details and this would imply dealing with the the problem in all its complexity.

Paradigms on which problems are solved are based on models that are simplified assumptions that reflect limitations as to the applicability of the model (whether a computer system, program an engine, etc.). By hiding complexity we are limiting those other options to be handled and decided by "experts". Also by assuming "experts" have correctly abstracted the problem to its core components is a strong assumption to be made under any scenario.

The counterpoint can always be made by assuming infinite regress, but this counterpoint is more than likely flawed because infinite regress based on infinities is a construct introduced by some mathematical models and not real world models.

While there is no easy way of dealing with complexity, doing so by abstraction in my opinion is not the best way to go. There are some compelling arguments as counterpoints and that is why I brought this question to this forum and put forth both sides of the argument.

I'm still having trouble understanding the purpose of this discussion from the author's point of view. I have a feeling there's an axe to grind but have trouble figuring it out. I don't see "both sides of the argument" here.

I will venture to describe the hiding of complexity in another context:

In approximation theory, we seek to find a "good enough" fit to data or a complicated function. Before there were easily accessible computers, mere humans needed to be able to calculate efficiently so the approximating functions that were acceptable were simple ones. As such, they could easily "hide" the accepted complexity of the original function but were nonetheless useful. In finance, the rule of 72 is an example.

Part of the discussion needs to be "how much abstraction is acceptable?" One would probably not use the rule of 72 to calculate their loan payments.

Fed,

You are assuming a mathematical analysis while the question is geared specifically towards computer systems (as the question clearly states). You are pointing to approximation theory which is a branch of mathematics and not computer systems. My bringing the point of complexity theory in mathematics is to draw a parallel towards how computer systems theory can improve by looking at how other branches of knowledge are tackling the problems.

Note:Your point of approximation theory assumes that the closed solution is the baseline for solving mathematics. This view is based on mathematical abstraction assumption that is not the only way to view mathematics.

Hiding complexity go a long way to aid usage. Client users are much concerned about the the interactive - ability of a system to meet their goal or interest. It's of course the experts interest to keep complexity confined as well put by @veerendra

I did say the example was in another context.

I did not say anything about closed solutions but would include them of course.

Perhaps we could use Matlab as an example of hiding complexity then? It surely does that, particularly for some built-in functions. What can be said in context about that? Is this not a useful example for which to discuss the points (which are still unclear to me).